Goal Orientation, Self-Efficacy, and "Online Measures" in Intelligent Tutoring Systems

Goal Orientation, Self-Efficacy, and “Online Measures” in Intelligent Tutoring Systems Stephen E. Fancsali 1 (sfancsali@carnegielearning.com) Matthew L. Bernacki 2 (matt.bernacki@unlv.edu) Timothy J. Nokes-Malach 3 (nokes@pitt.edu) Michael Yudelson 1 (myudelson@carnegielearning.com) Steven Ritter 1 (sritter@carnegielearning.com) Carnegie Learning, Inc., 437 Grant Street, Suite 918; Pittsburgh, PA 15219 USA Department of Educational Psychology & Higher Education, University of Nevada, Las Vegas; Las Vegas, NV 89154 USA Department of Psychology and Learning Research and Development Center, University of Pittsburgh; Pittsburgh, PA 15260 USA Abstract While goal orientation and related factors like learner self- efficacy are of great interest to learning science researchers, some voice concerns regarding the measurement of such factors using self-report questionnaires. To address these concerns, recent work has explored the use of behavioral indicators like hint-seeking and glossary use in intelligent tutoring systems like Carnegie Learning’s Cognitive Tutor ® (CT) as alternative, “online” measures of goal orientation. We re-examined this approach by measuring 273 CT users’ achievement goals and self-efficacy judgments via embedded questionnaires and their hint-seeking and glossary use via log data. Using graphical causal models and linear structural equation models to observe structural relationships among goal orientations, self-efficacy, behaviors, and learning outcomes, we found that tracing orientations via “online measures” is more nuanced than perhaps previously appreciated. We describe complex relations observed in the model among motivations, behaviors, and outcomes and discuss the implications for the online measurement of motivation. Keywords: Goal Orientation; Motivation; Self-Efficacy; Non-Cognitive Factors; Intelligent Tutoring Systems; Structural Equation Models; Graphical Causal Models. Introduction One well-studied aspect of motivation for learning focuses on individuals' achievement goals when approaching a learning task. Dweck (1986) provides a distinction between mastery and performance goal orientations. Learners have a mastery goal orientation when they seek to understand (i.e., master) a particular task or domain of interest. Those who seek to perform better relative to others have a performance goal orientation. Later work added another dimension of variation: a “valence” of either approaching success or avoiding failure (Elliot & McGregor, 2001). Learner goals corresponding to a mastery approach are those aimed at developing competence with respect to a task or learning objective, perhaps over a previous personal level of competence or other self-imposed criterion for task-mastery (Ames, 1992; Elliot, 1999); performance approach goals seek to demonstrate competence by outperforming peers. Learners who endorse performance avoidance goals seek to demonstrate that they are not any less competent than peers. Self-report questionnaires are commonly used to measure goal orientation. Generally, questionnaires are provided to learners either before or after a learning task. However, goal orientation can change dynamically as learners progress through a learning experience and have been shown to vary over longer time periods (e.g., a semester; Richardson, 2004; Fryer & Elliot, 2007; Muis & Edwards, Consequently, recent work (Otieno, Schwonke, Salden, & Renkl, 2013) suggests that, given changing or state-like aspects of goal orientations, fine-grained, “online” measures of goal orientation (i.e., those extracted from software log “traces”) may be a fruitful supplement to, and a potentially better measure than, questionnaire data in learning environments like intelligent tutoring systems (ITSs). While we agree that developing and validating appropriate “online” measures of goal orientation as well as other motivational, metacognitive, and cognitive processes is an important line of research, we suggest that relatively simple, proposed online measures may not provide a sufficiently nuanced assessment of underlying phenomena and may conflate a motivational construct with a behavior resulting from one or more motivations. We considered data from a study conducted by the second and third authors that addresses state-like aspects of goal orientation using online, in-tutor (i.e., between units of mathematics content) questionnaires in Carnegie Learning's Cognitive Tutor ® (CT) (Carnegie Learning, 2012; Ritter, Anderson, Koedinger, & Corbett, 2007) ITS for mathematics. We adopted a path analytic approach using structural equation models to investigate relationships among a variety of self-reports of students’ motivation, online measures of students’ behavior in and interaction with the CT, and performance outcomes. We specified a structural equation model by learning a set of qualitative causal structures consistent with both data and background knowledge using the framework of semi-automated, algorithmic search for graphical causal models (Spirtes, Glymour, & Scheines 2000; Pearl, 2009). We evaluate the proposal of Otieno, et al. (2013) that hint and glossary use in the CT may serve as online indicators of student motivation (i.e., goal orientation) and found that their proposed mapping of traced behavior to motivational