Concept-Map Based Assessment: On Possible Sources of Sampling Variability.

A concept-map assessment consists of a task that elicits structured knowledge, a response format, and a scoring system. Variation in tasks, response formats, and scoring systems produce different mapping techniques that may elicit different knowledge representations, posing construct-interpretations challenges. This study examined two mapping techniques: (1) students generated the 10 concepts from chemistry to construct a map; and (2) the assessor provided a list of 10 concepts. Two concept-lists were randomly sampled from the domain to examine the effect of concept sampling on map scores. Forty high school students, two teachers, and one expert participated. Results indicate that: (1) the two mapping techniques were statistically equivalent; (2) students' concept-map scores generalized across samples of concepts; (3) concept maps could be reliably scored, even though they involved complex judgments; and (4) multiple-choice test and concept maps measure somewhat different aspects of science knowledge. Appendixes contain a discussion of compiling the list of concepts, a list of concepts considered for the three conditions, a sample of instructions, and a matrix of the relations between pairs of concepts. (Contains 9 tables and 32 references.) (Author/SLD) ******************************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. ******************************************************************************** Concept-Map Based Assessment 1 Running Head: CONCEPT-MAP BASED ASSESSMENT Concept-Map Based Assessment: On Possible Sources of Sampling Variability Maria Araceli Ruiz-Primo and Richard J. Shavelson Stanford University PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS BEEN GRANTED BY pit 18i,Vczo 1 TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) March 29, 1997 U.S. DEPARTMENT OF EDUCATION Office 1st Educational Research and Improvement EDUCATIONAL RESOURCES INFORMATION CEt hNTaEs beR(EeRn IC)p FThis docum roduced as received from the person or organization originating it. 1:1 Minor changes have been made to improve reproduction quality. Points of view or opinions stated in this document do not necessarily represent official OERI position or policy. * The work reported herein was supported by the Center for Research on Evaluation, Standards, and Student Testing under the Educational Research and Development Centers Program PR/Award Number R305B60002, as administered by the Office of Educational Research and Improvement, U.S. Department of Education. The findings and opinions expressed in this report reflect those of the authors ant not necessarily those of the funding agencies. The authors are deeply grateful to Susan Elise Shultz, for her help in collecting data and scoring students maps, and Dr. Pinchas Tamir, for his valuable comments. Concept-Map Based Assessment 2 Abstract A concept-map assessment consists of a task that elicits structured knowledge, a response format and a scoring system. Variation in tasks, response formats, and scoring systems produce different mapping techniques that may elicit different knowledge representations, posing constructinterpretations challenges. This study examined two mapping techniques: (1) students generated the 10 concepts from chemistry to construct a map, and (2) assessor provided a list of 10 concepts. Two concept-lists were randomly sampled from the domain to examine the effect of concept sampling on map scores. Forty high-school students, two teachers and one expert participated. Results indicated that: (a) the two mapping techniques were statistically equivalent; (b) students concept-map scores generalized across samples of concepts; (c) concept maps could be reliably scored, even though they involved complex judgments; and (d) multiple-choice test and concept map measure somewhat different aspects of science knowledge.A concept-map assessment consists of a task that elicits structured knowledge, a response format and a scoring system. Variation in tasks, response formats, and scoring systems produce different mapping techniques that may elicit different knowledge representations, posing constructinterpretations challenges. This study examined two mapping techniques: (1) students generated the 10 concepts from chemistry to construct a map, and (2) assessor provided a list of 10 concepts. Two concept-lists were randomly sampled from the domain to examine the effect of concept sampling on map scores. Forty high-school students, two teachers and one expert participated. Results indicated that: (a) the two mapping techniques were statistically equivalent; (b) students concept-map scores generalized across samples of concepts; (c) concept maps could be reliably scored, even though they involved complex judgments; and (d) multiple-choice test and concept map measure somewhat different aspects of science knowledge.