Using an Instrument Blueprint to Support the Rigorous Development of New Surveys and Assessments in Engineering Education

Many sound methods exist for creating the items or questions that make up educational surveys and assessments. These methods include the use of content experts, reviews of existing instruments, and lists of behaviors and descriptors commonly associated with the construct(s) we wish to assess. Unfortunately, however, item creation sometimes becomes overly dependent upon a researcher’s personal attitudes about the construct(s) being tested, or on “borrowing” items from other instruments that may or may not be sound measures of the construct(s) of interest. These risks are particularly likely for new researchers in engineering education, who may have little experience with best practices in social science research. One way to support best practices in the development of new surveys and assessments is to use an instrument blueprint to guide the creation of items, as well as the collection of validity evidence. This paper outlines a process for instrument blueprint creation and content validation to help support best practices in educational assessment. Based on Messick’s unified theory of validity, the instrument blueprint includes a process for item construction that incorporates multiple resources, including: (1) the views of content experts; (2) research from the relevant domain of interest; (3) reviews of existing instruments; and (4) the expertise of the research team. This paper uses the development of a new instrument to measure engineering innovativeness as an illustrative example of the blueprinting process. Our new instrument will assess 20 characteristics of innovative engineers as identified by in-depth studies of expert engineering innovators in previous research. This work highlights the positive impact of using a systematic process for item construction to transform current methods of assessment in engineering education.