Construct Validity of Free-Response and Machine-Scorable Forms of a Test.

A persistent issue in ability testing concerns the importance of the kind of response required by a test item. Critics of multiple-choice testing argue that certain important aspects of ability can only be measured with open-ended tests, in which the examinee must generate and organize his answers; supporters claim that any ability that is clearly specified can be assessed by "objective" procedures, in which the correct answer need only be chosen from among alternatives presented. Psychometric researchers have rarely sought the kind of evidence needed to resolve this question. Studies of the ability of various tests to predict criteria such as academic grade-point average are not sufficient; such studies can indicate whether one test is a more efficient predictor of a "criterion" than another, but, since the criteria are themselves complex, not whether one is measuring something different from the other. Relevant evidence can be obtained through studies of construct validity. Messick (1975) defined construct validation as "the process of marshaling evidence in the form of theoretically relevant empirical relations to support the inference that [a test score] has a particular meaning." Messick went on to say that "test validation in the construct framework is integrated with hypothesis testing and with all of the philosophical and empirical means by which scientific theories are evaluated." In other words, a study of the construct validity of a test is equivalent to a scientific investigation of the cognitive processes involved in taking a test. The value of experimental and observational approaches to construct validation is demonstrated by progress made in the last couple of decades by cognitive psychologists in studies of such activities as recall, recognition, verification, problem solving, rule induction, and reading (Estes, 1978; Gregg, 1974; Newell & Simon, 1972; Sternberg, 1977). This work has succeeded in describing mental operations with a degree of precision far greater than that available from traditional correlational research on aptitude and achievement tests. But properly designed psychometric studies may also throw light on the processes involved in cognitive activities (Hunt, Frost, & Lunneborg, 1973; Hunt, Lunneborg, & Lewis, 1975; Lunneborg, 1978; Pellegrino & Glaser, 1979). A psychometric comparison of the construct validity of two tests would require one to (1) develop hypotheses about the cognitive skills and personality traits that might