Assessment of Thinking Levels in Students' Answers

Authors' Note: This research was supported in part by a grant to J.J. Pear from the Social Sciences and Humanities Research Council of Canada. D.E. Crone-Todd was supported by a fellowship from the Social Sciences and Humanities Research Council of Canada. The authors gratefully acknowledge Ms. Sabrina Berry's assistance with this research project. Abstract Having first developed a method, based on Bloom's taxonomy (1956), for assessing the thinking levels required by study questions in computer-mediated courses (Crone-Todd, Pear & Read, 2000), we developed a method for assessing the levels at which students answer the questions. Reliability measures between two independent assessment groups were high (i.e., > 80%). The assessment procedure can serve diagnostic and research purposes in determining how to enable students to increase their thinking levels in post-secondary courses. ********** Assessment of Thinking Levels in Students' Answers One of the most important goals of post-secondary education is to promote the use of critical, or higher-order, thinking skills. To this end, educators must find ways to identify, teach, and encourage the use of these skills in their courses. One of the largest hurdles in this process is developing a precise operational definition, or set of definitions, for what is meant by "higher-order thinking". There is, however, a lack of consensus concerning the definition of this construct. For example, higher-order thinking may be "reasoned argumentation" (Newman, 1991a, b), comparing elements in terms of sameness (Carnine, 1991), application of concepts or principles (Hohn, Gallagher, & Byrne, 1990; Semb & Spencer, 1976), making discipline-related judgments that are effective (Paul & Heaslip, 1995), or argumentation that is systematic and active (Mayer & Goodchild, 1990). It seemed to us that all of these definitions include various components of what is considered "higher-order thinking", or thinking that requires combining elements in different ways than those provided in a textbook or other course materials. A set of definitions that appears to incorporate all of the definitions above is Bloom's (1956) taxonomy of objectives in the cognitive domain. The taxonomy, which incorporates behavioral definitions of cognitive processes, has been used in a variety of educational settings. Despite its popularity, however, those using the taxonomy for research purposes have encountered problems with its reliability (e.g., Calder, 1983; Gierl, 1997; Kottke & Schuster, 1990; Roberts, 1976; Seddon, 1978; Seddon, Chokotho, & Merritt, 1981). Recently, Crone-Todd, Pear, & Read (2000) used a modified version of Bloom's (1956) taxonomy in the cognitive domain to identify the thinking levels required by study questions in a computer-aided personalized system of instruction (CAPSI) course. The purpose of the study was to begin the development of a more reliable measure of higher-order thinking in CAPSI-taught using guided study questions (e.g., Pear & Crone-Todd, 1999; http://home.cc.umanitoba.ca/~capsi) than had been previously reported in the literature. Following the taxonomy, the thinking levels were: (a) Level 1 - Knowledge, (b) Level 2 - Comprehension, (c) Level 3 - Application, (d) Level 4 - Analysis, (e) Level 5 - Synthesis, and (f) Level 6 - Evaluation. Briefly, in the modified taxonomy, Level 1 corresponds to rote learning, Level 2 involves the ability to state an answer in one's own words, Level 3 is the ability to apply what one has learned to new problems or situations, Level 4 is the ability to break down concepts into smaller components, Level 5 is the ability to combine concepts to create new knowledge, and Level 6 is the ability to rationally argue or discuss a position with regard to a given topic. Levels 1 and 2 may be considered lower-order thinking (because they do not involve generation of new concepts or knowledge), while levels 3 through 6 may be consider higher-order thinking (see Crone-Todd et. …