How Many Times Should a Pedagogical Agent Simulation Model Be Run?

When using simulation modeling to explore pedagogical phenomena, there are several issues a designer/practitioner should consider. One of the most important decisions has to do with determining how many runs of a simulation to perform in order to be confident in the results produced by the simulation [1]. With a deterministic model, a single simulation run is adequate. This issue becomes more challenging when part of the simulation model is based on stochastic elements. One of the solutions that has been used to address this challenge in other research communities is the use of Monte Carlo simulation [2]. Within the AIED research community, however, this question of how many times should a pedagogical simulation model be run to produce predictions in which the designer can have confidence has received surprisingly little attention. The aim of this paper is to explore this issue using a pedagogical simulation model, SimDoc, designed to explore longer term mentoring issues [3]. In particular, we demonstrate how to run this simulation model over many iterations until the accumulated results of the iteration runs reach a statistically stable level that matches real world performance but also has appropriate variability among the runs. We believe this approach generalizes beyond our simulation environment and could be applied to other pedagogical simulations and would be especially useful for medium and high fidelity simulations where each run may take a long time.

[1]  G. McCalla,et al.  Using the Ecological Approach to Create Simulations of Learning Environments , 2013, AIED.

[2]  Alexander S. Yeh,et al.  More accurate tests for the statistical significance of result differences , 2000, COLING.

[3]  David Edgar K. Lelei,et al.  Exploring the Issues in Simulating a Semi-Structured Learning Environment: The SimGrad Doctoral Program Design , 2015, AIED Workshops.

[4]  Kurt VanLehn,et al.  Applications of simulated students: an exploration , 1994 .

[5]  Gordon McCalla,et al.  How to Use Simulation in the Design and Evaluation of Learning Environments with Self-directed Longer-Term Learners , 2018, AIED.

[6]  H. Chad Lane,et al.  Preface to the IJAIED 25th Anniversary Issue, Part 2 , 2016, International Journal of Artificial Intelligence in Education.

[7]  Gerhard Weber,et al.  Individual selection of examples in an intelligent learning environment , 1996 .

[8]  Fabiano Azevedo Dorça.  Implementation and use of Simulated Students for Test and Validation of new Adaptive Educational Systems: a Practical Insight , 2015, International Journal of Artificial Intelligence in Education.

[9]  Wouter van Joolingen,et al.  Design and implementation of simulation-based discovery environments: the SMISLE solution , 1996 .

[10]  Carolyn Penstein Rosé,et al.  Building a Conversational SimStudent , 2012, ITS.

[11]  Ryan Shaun Joazeiro de Baker,et al.  Using a Model for Learning and Memory to Simulate Learner Response in Spaced Practice , 2017, AIED.

[12]  Chao-Lin Liu,et al.  A Simulation-Based Experience in Learning Structures of Bayesian Networks to Represent How Students Learn Composite Concepts , 2008, Int. J. Artif. Intell. Educ..

[13]  Terence John Gatfield,et al.  An Investigation into PhD Supervisory Management Styles: Development of a dynamic conceptual model and its managerial implications , 2005 .

[14]  Michel C. Desmarais,et al.  A Bayesian Student Model without Hidden Nodes and its Comparison with Item Response Theory , 2005, Int. J. Artif. Intell. Educ..

[15]  Fuhua Lin,et al.  Simulated Learners for Testing Agile Teaming in Social Educational Games , 2015, AIED Workshops.

[16]  Holger Hermanns,et al.  Simulation and Statistical Model Checking for Modestly Nondeterministic Models , 2012, MMB/DFT.

[17]  Trevor Heath,et al.  A Quantitative Analysis of PhD Students' Views of Supervision , 2002 .

[18]  Johanna D. Moore,et al.  BEETLE II: Deep Natural Language Understanding and Automatic Feedback Generation for Intelligent Tutoring in Basic Electricity and Electronics , 2014, International Journal of Artificial Intelligence in Education.

[19]  Frank E. Ritter,et al.  Determining the number of simulation runs: Treating simulations as theories by not sampling their behavior , 2011 .

[20]  J. Booth,et al.  Monte Carlo Approximation of Bootstrap Variances , 1998 .

[21]  S. Haneuse,et al.  On the Assessment of Monte Carlo Error in Simulation-Based Statistical Analyses , 2009, The American statistician.

[22]  H. Chad Lane,et al.  The Next 25 Years: How Advanced Interactive Learning Technologies will Change the World , 2016 .

[23]  Gordon I. McCalla,et al.  Exploring Through Simulation an Instructional Planner for Dynamic Open-Ended Learning Environments , 2015, AIED.

[24]  Graham Currie,et al.  How Many Simulation Runs are Required to Achieve Statistically Confident Results: A Case Study of Simulation-Based Surrogate Safety Measures , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.