Development and Validation of a Behavioral Measure of Metacognitive Processes ( BMMP )

750-1000 words) Based on the assumption that learning is a generative process involving human cognitive, metacognitive and motivational processes (Wittrock, 1990), the focus of learning is not only on the final outcome (i.e. performance outcome) but also the process of learning (Mayer, 1999). The interactive nature of computer-based learning environments impose challenges to learners by leaving the locus of control mainly in the hands of the learners (Clarebout & Elen, 2004; Hill & Hannafin, 2001; Swaak & de Jong, 2001), whereby learners need to be able to manage how, what and when to use appropriate resources to guide them through effective learning process (Hill & Hannafin, 2001). Further, exploratory learning environments impose added burden on the learners to control their learning process according to the learning tasks, learning contexts and their own cognitive level and proceed through the steps to identify problem, develop hypothesis, explore, observe and analyze provided information (de Jong & van Joolingen, 1998), prompting some researchers to call these environments ineffective (Kirschner, Sweller, & Clark, 2007). Strategies employed by learners to direct their learning process are thought to determine their learning outcome and are studied as metacognitive processes. Metacognition, i.e., the appropriate use of metacognitive and self-regulatory strategies, has been found to influence the effectiveness of learning as well as the use of resources provided in the learning environment (Aleven & Koedinger, 2000; du Boulay, et al., 1999; Clarebout, et al., 2004; Gräsel, et al., 2001; Hill & Hannafin, 2001; Oliver & Hannfin, 2000). Although metacognitive engagement during learning is considered an important predictor of learning outcome, measures of metacognitive processes have largely relied on verbal reports such as self-report surveys, interviews or concurrent verbal protocols. While verbal data is often used as a method of assessing cognitive processes in different experimental settings, the validity and accuracy of verbal data is long disputed. Verbal reports based on introspection are often considered useful for discovery or identification of psychological processes rather than as a method of verification, which requires more objective measurement for validation (Lashley, 1923). The present study In the present study, conducted as part of a three-year research grant Molecules & Minds: Optimizing Simulations for Chemistry Education, funded by the US Department of Education’s Institute of Education Sciences (IES), we explored the use of a behavioral measure of metacognitive process (BMMP) during the exploration of a science simulation that is based on the analysis of log file data collected during simulation exploration. Log analysis is thought to be a method that provides effective means of assessing user behavior in computer-based learning environment (Lawless & Brown, 1997; Leutner & Plass, 1998) as it allows for a non-intrusive observation of learner’s actions that can be used to examine the underlying cognitive processes such as knowledge acquisition strategies, information search strategies, and problem solving processes (Guthrie and Dreher, 1990; Lawless, & Brown, 1997). The BMMP and the self-report measure of metacognitive control were examined to identify a possible mediating role of metacognitive processes on the learning effects of exploratory computer simulation. Prior knowledge, widely researched as a learner characteristic determinant of learning outcome (Dochy, Segers, & Buehl, 1999) and found to positively influence performance (Bloom, 1976; Dochy, 1992; Tobias, 1994), was also examined for its influence on the learning performance. Method Eighty-eight students from public high school in New York City participated in the research as they studied Kinetic Theory of Heat using an interactive computer simulation. Students’ prior knowledge and learning outcome were assessed using knowledge preand post-test administered before and after exploring the simulation, respectively. Selfregulatory strategies (8 items survey from MSLQ) were assessed through a self-report survey administered after the treatment. The BMMP was based on learner’s activity log recorded as they used the simulation. Each log file was analyzed for proportion of meaningful action (i.e. sequential testing and observation of at least two different values for an independent variable affecting the phenomenon of Kinetic Theory of Heat represented by the simulation) over total number of interaction with the simulation, and the proportion of variables interacted over total number of independent variables that can be explored through the simulation. The two proportions were added to obtain the BMMP score of metacognitive control for each subject. Table 1. Correlation of BMMP, self-regulation, prior knowledge, and learning outcomes BMMP Self-regulation Prior Knowledge Post-test Metacognitive Score 1 SelfRegulation -.063 1 Prior Knowledge .195* .119 1 Post.424** -.150 .184 1