Learning How to Construct Models of Dynamic Systems: An Initial Evaluation of the Dragoon Intelligent Tutoring System

Constructing models of dynamic systems is an important skill in both mathematics and science instruction. However, it has proved difficult to teach. Dragoon is an intelligent tutoring system intended to quickly and effectively teach this important skill. This paper describes Dragoon and an evaluation of it. The evaluation randomly assigned students in a university class to either Dragoon or baseline instruction that used Dragoon as an editor only. Among students who did use their systems, the tutored students scored reliably higher (p < .021, d = 1.06) on the post-test than the students who used only the conventional editor-based instruction.

[1]  Kurt VanLehn,et al.  Evaluation of a meta-tutor for constructing models of dynamic systems , 2013, AIED 2013.

[2]  K. VanLehn,et al.  Cognitive skill acquisition. , 1996, Annual review of psychology.

[3]  John R. Anderson,et al.  Illustrating Principled Design: The Early Evolution of a Cognitive Tutor for Algebra Symbolization , 1998, Interact. Learn. Environ..

[4]  Mitchell J. Nathan.  Knowledge and Situational Feedback in a Learning Environment for Algebra Story Problem Solving , 1998, Interact. Learn. Environ..

[5]  Asunción Gómez-Pérez,et al.  DynaLearn - Engaging and Informed Tools for Learning Conceptual System Knowledge , 2009, AAAI Fall Symposium: Cognitive and Metacognitive Educational Systems.

[6]  Cleotilde Gonzalez,et al.  Understanding the building blocks of dynamic systems , 2007 .

[7]  Peter Norvig,et al.  Artificial Intelligence: A Modern Approach , 1995 .

[8]  Timothy Teo,et al.  The role of model building in problem solving and conceptual change , 2011, Interact. Learn. Environ..

[9]  Gautam Biswas,et al.  Designing Learning by Teaching Agents: The Betty's Brain System , 2008, Int. J. Artif. Intell. Educ..

[10]  Gautam Biswas,et al.  Integrating computational thinking with K-12 science education using agent-based computation: A theoretical framework , 2013, Education and Information Technologies.

[11]  Cleotilde González,et al.  Why don ’ t well-educated adults understand accumulation ? A challenge to researchers , educators , and citizens , 2008 .

[12]  T. Jong,et al.  Supporting simulation-based learning; the effects of model progression and assignments on definitional and intuitive knowledge , 1998 .

[13]  Daniel Kainz,et al.  Can Students Learn Stock-Flow-Thinking? An Empirical Investigation , 2002 .

[14]  S. Russel and P. Norvig,et al.  “Artificial Intelligence – A Modern Approach”, Second Edition, Pearson Education, 2003. , 2015 .

[15]  Kenneth D. Forbus,et al.  Distributed Coaching for an Intelligent Learning Environment , 1998 .

[16]  Hugh F. Cline,et al.  Modeling and Simulation in the Secondary School Curriculum: The Impact on Teachers , 1994, Interact. Learn. Environ..

[17]  E. Soloway,et al.  Causal Model Progressions as a Foundation for Intelligent Learning Environments , 1990 .

[18]  Elwin R. Savelsbergh,et al.  The effect of external representation on constructing computer models of complex phenomena , 2003 .

[19]  Kurt VanLehn,et al.  From behavioral description to a pattern-based model for intelligent tutoring systems , 2011, PLoP '11.

[20]  Joseph Krajcik,et al.  A learner-centered tool for students building models , 1996, CACM.

[21]  M. Resnick,et al.  Thinking in Levels: A Dynamic Systems Approach to Making Sense of the World , 1999 .

[22]  Erling Moxnes,et al.  Not only the tragedy of the commons: misperceptions of feedback and policies for sustainable development , 2000 .

[23]  Albert T. Corbett,et al.  Student Question-Asking Patterns in an Intelligent Algebra Tutor , 2004, Intelligent Tutoring Systems.

[24]  V. Aleven,et al.  Help Seeking and Help Design in Interactive Learning Environments , 2003 .

[25]  Barbara A. Crawford,et al.  Supporting prospective teachers' conceptions of modelling in science , 2004 .

[26]  Johanna Weiss.  Classroom Dynamics Implementing A Technology Based Learning Environment , 2016 .

[27]  Ton de Jong,et al.  Using Co-Lab to build System Dynamics models: Students' actions and on-line tutorial advice , 2009, Comput. Educ..

[28]  Ton de Jong,et al.  What Can Be Learned from Computer Modeling? Comparing Expository and Modeling Approaches to Teaching Dynamic Systems Behavior , 2012 .

[29]  Murat Akkuş.  The Common Core State Standards for Mathematics , 2015 .

[30]  Kenneth D. Forbus,et al.  Using Qualitative Physics to Build Articulate Software for Thermodynamics Education: A Preliminary Report , 1994, Interact. Learn. Environ..

[31]  Kurt VanLehn,et al.  Model construction as a learning activity: a design space and review , 2013, Interact. Learn. Environ..

[32]  Joseph Krajcik,et al.  The design of guided learner-adaptable scaffolding in interactive learning environments , 1996, CHI.

[33]  Kurt VanLehn,et al.  Studying, Teaching and Applying Sustainability Visions Using Systems Modeling , 2014 .

[34]  Wouter R. van Joolingen,et al.  Students' reasoning during modeling in an inquiry learning environment , 2005, Comput. Hum. Behav..

[35]  Lars Bollen,et al.  Learning from erroneous models using SCYDynamics , 2014 .

[36]  Sandra Katz,et al.  Toward More Robust Learning of Physics via Reflective Dialogue Extensions , 2009 .

[37]  Ton de Jong,et al.  Co-Lab: research and development of an online learning environment for collaborative scientific discovery learning , 2005, Comput. Hum. Behav..

[38]  J. Michael Spector,et al.  Integrated and Holistic Perspectives on Learning, Instruction and Technology , 2000 .

[39]  Kurt VanLehn,et al.  Defining the Behavior of an Affective Learning Companion in the Affective Meta-tutor Project , 2013, AIED.

[40]  Helen M. Doerr,et al.  Stella ten years later: A review of the literature , 1996, Int. J. Comput. Math. Learn..

[41]  Kurt VanLehn,et al.  Eliminating the Gap between the High and Low Students through Meta-cognitive Strategy Instruction , 2008, Intelligent Tutoring Systems.

[42]  Kurt VanLehn,et al.  Meta-Cognitive Strategy Instruction in Intelligent Tutoring Systems: How, When, and Why , 2010, J. Educ. Technol. Soc..

[43]  Richard Boohan,et al.  Children and computer modelling: making worlds with WorldMaker , 1995, WCCE.

[44]  Joseph Krajcik,et al.  Secondary Students' Dynamic Modeling Processes: Analyzing, Reasoning About, Synthesizing, and Testing Models of Stream Ecosystems , 1997 .

[45]  Alexander Repenning,et al.  AgentSheets: End-User Programmable Simulations , 2000, J. Artif. Soc. Soc. Simul..

[46]  Carolyn Penstein Rosé,et al.  CycleTalk: Toward a Dialogue Agent That Guides Design with an Articulate Simulator , 2004, Intelligent Tutoring Systems.

[47]  Eric K. Neumann,et al.  An Object-Based Modeling Tool for Science Inquiry , 1999 .

[48]  Sandra P. Marshall,et al.  Story Problem Solver: A Schema-Based System of Instruction. , 1989 .

[49]  Kenneth D. Forbus,et al.  Qualitative Modeling in Education , 2004, AI Mag..

[50]  Nikolaos Avouris,et al.  Modellingspace: interaction design and architecture of a collaborative modelling environment , 2003 .

[51]  John S. Kinnebrew,et al.  A science learning environment using a computational thinking approach , 2012, ICCE 2012.

[52]  Albert T. Corbett,et al.  Student Questions in a Classroom Evaluation of the ALPS Learning Environment , 2005, AIED.

[53]  Marcelo Resende Thielo,et al.  Students modelling environmental issues , 1997, J. Comput. Assist. Learn..

[54]  Ryan Shaun Joazeiro de Baker,et al.  Adapting to When Students Game an Intelligent Tutoring System , 2006, Intelligent Tutoring Systems.

[55]  Christopher K. Riesbeck,et al.  Fostering reflection with Socratic tutoring software: results of using inquiry teaching strategies with web-based HCI techniques , 2004 .

[56]  Ard W. Lazonder,et al.  Finding Out How They Find It Out: An empirical analysis of inquiry learners’ need for support , 2010 .

[57]  David McArthur.  Algebraic Thinking Tools: Supports for Modeling Situations and Solving Problems in Kids' Worlds. , 1989 .

[58]  Harvey Mellar,et al.  Educational tools for computational modelling , 1993 .

[59]  U. Wilensky,et al.  Complex Systems in Education: Scientific and Educational Importance and Implications for the Learning Sciences , 2006 .

[60]  Lars Bollen,et al.  Validating and Optimizing the Effects of Model Progression in Simulation-Based Inquiry Learning , 2012 .

[61]  Inge Molenaar,et al.  Computer Based Modeling Tasks: the Role Of External Represen- tation , 2006 .

[62]  Peter Norvig,et al.  Artificial intelligence - a modern approach, 2nd Edition , 2003, Prentice Hall series in artificial intelligence.

[63]  Herbert H. Clark,et al.  Grounding in communication , 1991, Perspectives on socially shared cognition.

[64]  Kurt VanLehn,et al.  The Affective Meta-Tutoring Project: How to motivate students to use effective meta-cognitive strategies , 2011, ICCE 2011.

[65]  Kathleen Hogan,et al.  Cognitive Comparisons of Students' Systems Modeling in Ecology , 2001 .

[66]  Bert Bredeweg,et al.  Context-Dependent Help for the DynaLearn Modelling and Simulation Workbench , 2011, AIED.

[67]  Tak-Wai Chan,et al.  Exploring the Design of Computer Supports for Reciprocal Tutoring , 1997 .

[68]  Pat Langley,et al.  An interactive environment for the modeling and discovery of scientific knowledge , 2006, Int. J. Hum. Comput. Stud..

[69]  Ton de Jong,et al.  Scientific Discovery Learning with Computer Simulations of Conceptual Domains , 1998 .

[70]  Jon Ogborn,et al.  Sixth form students' ability to engage in computational modelling , 1994 .

[71]  Krystyna Anne Stave,et al.  Assessing the Effectiveness of Systems Thinking Interventions in the Classroom , 2008 .

[72]  Kurt VanLehn,et al.  Using HCI Task Modeling Techniques to Measure How Deeply Students Model , 2013, AIED Workshops.

[73]  Sandra Katz,et al.  Out of the Lab and into the Classroom: An Evaluation of Reflective Dialogue in Andes , 2007, AIED.

[74]  Floris Linnebank,et al.  Progressive Knowledge Representations for Learning Conceptual Knowledge of System Behaviour , 2010 .

[75]  Daniel L. Schwartz,et al.  Interactive Metacognition: Monitoring and Regulating a Teachable Agent , 2009 .

[76]  Kurt VanLehn,et al.  The Behavior of Tutoring Systems , 2006, Int. J. Artif. Intell. Educ..

[77]  Steven J. Stratford.  A review of computer-based model research in precollege science classrooms , 1997 .

[78]  Lars Bollen,et al.  SimSketch: Multiagent Simulations Based on Learner-Created Sketches for Early Science Education , 2013, IEEE Transactions on Learning Technologies.

[79]  F. Reif,et al.  Teaching scientific thinking skills: Students and computers coaching each other , 1999 .

[80]  Neil T. Heffernan,et al.  Expanding the Model-Tracing Architecture: A 3rd Generation Intelligent Tutor for Algebra Symbolization , 2008, Int. J. Artif. Intell. Educ..

[81]  K. Koedinger,et al.  Improving students’ help-seeking skills using metacognitive feedback in an intelligent tutoring system , 2011, Learning and Instruction.

[82]  Sandra Katz,et al.  Going Beyond the Problem Given: How Human Tutors Use Post-Solution Discussions to Support Transfer , 2003, Int. J. Artif. Intell. Educ..

[83]  Robert E. Wood,et al.  The effects of feedback format on dynamic decision making , 2002 .

[84]  Magnus Haake,et al.  A Teachable-Agent Arithmetic Game's Effects on Mathematics Understanding, Attitude and Self-efficacy , 2011, AIED.

[85]  Kate Thompson,et al.  Patterns of use of an agent-based model and a system dynamics model: The application of patterns of use and the impacts on learning outcomes , 2010, Comput. Educ..

[86]  David F. Treagust,et al.  Students' understanding of the role of scientific models in learning science , 2002 .

[87]  Ton de Jong,et al.  The Integration of Computer Simulation and Learning Support: An Example from the Physics Domain of Collisions , 1999 .

[88]  Quinn James,et al.  The Effects of Simulation Complexity and Hypothesis-Generation Strategy on Learning , 1994 .

[89]  J. Sterman,et al.  Cloudy Skies: Assessing Public Understanding of Global Warming , 2002 .

[90]  Leo C. Ureel,et al.  VModel: A Visual Qualitative Modeling Environment for Middle-school Students , 2004, AI Mag..

[91]  Daniel L. Schwartz,et al.  Teachable Agents and the Protégé Effect: Increasing the Effort Towards Learning , 2009 .

[92]  Floris Linnebank,et al.  Garp3 - Workbench for qualitative modelling and simulation , 2009, Ecol. Informatics.

[93]  Joseph Krajcik,et al.  Making Dynamic Modeling Accessible to Precollege Science Students , 1994, Interact. Learn. Environ..

[94]  Gautam Biswas,et al.  LEARNING BY TEACHING: A NEW AGENT PARADIGM FOR EDUCATIONAL SOFTWARE , 2005, Appl. Artif. Intell..

[95]  Chris Dede,et al.  Model-Based Teaching and Learning with BioLogica™: What Do They Learn? How Do They Learn? How Do We Know? , 2004 .

[96]  B. White.  ThinkerTools: Causal Models, Conceptual Change, and Science Education , 1993 .

[97]  Kenneth D. Forbus.  The role of qualitative dynamics in naive physics , 1985 .

[98]  Mitchell J. Nathan,et al.  A theory of algebra-word-problem comprehension and its implications for the design of learning environments. , 1992 .

[99]  Min Chi,et al.  Adaptive Expertise as Acceleration of Future Learning A Case Study , 2011 .

[100]  Stephen M. Alessi,et al.  Building Versus Using Simulations , 2000 .

[101]  Jac A. M. Vennix,et al.  Effect of system dynamics education on systems thinking inventory task performance , 2005 .

[102]  John D. Sterman,et al.  Bathtub dynamics: initial results of a systems thinking inventory , 2000 .

[103]  Richard Stottler,et al.  A Meta-Cognitive Computer-based Tutor for High-School Algebra , 2003 .

[104]  Diana M. Fisher,et al.  Training System Modelers: The NSF CC-STADUS and CC-SUSTAIN Projects , 1999 .

[105]  Kurt VanLehn,et al.  The level up procedure: How to measure learning gains without pre- and post-testing , 2011, ICCE 2011.

[106]  Leo C. Ureel,et al.  Qualitative modeling for middle-school students , 2004 .

[107]  Kurt VanLehn,et al.  Effects of Dissuading Unnecessary Help Requests While Providing Proactive Help , 2005, AIED.

[108]  W. V. van Joolingen,et al.  The Difficult Process of Scientific Modelling: An analysis of novices' reasoning during computer‐based modelling , 2005 .

[109]  Richard E. Mayer,et al.  Frequency norms and structural analysis of algebra story problems into families, categories, and templates , 1981 .

[110]  Gautam Biswas,et al.  Measuring Self-Regulated Learning Skills through Social Interactions in a teachable Agent Environment , 2010, Res. Pract. Technol. Enhanc. Learn..

[111]  Barbara Y. White,et al.  Designing Computer Games to Help Physics Students Understand Newton's Laws of Motion , 1984 .

[112]  Neil T. Heffernan,et al.  Intelligent tutoring systems have forgotten the tutor: adding a cognitive model of human tutors , 1998, CHI Conference Summary.