Learning from playing with microworlds in policy making: An experimental evaluation in fisheries management

This study determines whether stakeholders learn from playing with microworlds. This is investigated through a case study of Belgian fisheries management. Policymakers, scientists and fishermen participated in a ''before-after with control group''-experiment in which they played with a microworld that aims at gaining insight into the long-term effect of policy instruments on the Belgian fisheries system. The outcome of this experiment indicates that using the microworld did not result in learning outcomes (i.e., changes in participants' subjective knowledge, attitude and behavioural intention towards policy instruments). This however, contradicts all stakeholders' reports that they had learned from the microworld and that they had confidence in the microworld and perceived the microworld to be valid. Hence, three alternative explanations for these results are discussed: (1) methodological issues blocked the detection of learning outcomes, (2) the way in which the microworld was administered did not result in learning outcomes, or (3) participants have not ''learned'' anything new from the microworld. Finally, the paper ends with discussing guidelines and further steps in evaluating learning from microworlds.

[1]  Cleotilde Gonzalez,et al.  The use of microworlds to study dynamic decision making , 2005, Comput. Hum. Behav..

[2]  A. J. Faria.  The Changing Nature of Business Simulation/ Gaming Research: A Brief History , 2001 .

[3]  John R. Dickinson,et al.  The Second Component to Experiential Learning: A Look Back At How ABSEL Has Handled the Conceptual and Operational Definitions of Learning , 1998 .

[4]  Hugh M. Cannon,et al.  Developments in Business Simulation and Experiential Learning, Volume 28, 2001 FIDELITY, VERIFIABILITY, AND VALIDITY OF SIMULATION: CONSTRUCTS FOR EVALUATION , 2001 .

[5]  George P. Richardson,et al.  A framework for evaluating systems thinking interventions: an experimental approach to mental health system change , 1997 .

[6]  Andy P. Field,et al.  Discovering Statistics Using SPSS , 2000 .

[7]  Simon Peck,et al.  Group Model Building: Facilitating Team Learning Using System Dynamics , 1996, J. Oper. Res. Soc..

[8]  John Washbush,et al.  A Review of Scholarship on Assessing Experiential Learning Effectiveness , 2004 .

[9]  Leslie H. Vincent,et al.  Objective and Subjective Knowledge Relationships: A Quantitative Analysis of Consumer Research Findings , 2009 .

[10]  J. L. McKenney,et al.  Influences on Learning in Simulation Games , 1966 .

[11]  R. Czaja.  Questionnaire Pretesting Comes of Age , 1998 .

[12]  J. Funke.  Microworlds based on linear equation systems: A new approach to complex problem solving and experimental results , 1993 .

[13]  Christiane Gresse von Wangenheim,et al.  Empirical evaluation of an educational game on software measurement , 2008, Empirical Software Engineering.

[14]  Stephen A. Stumpf,et al.  Microworlds and Simuworlds: Practice Fields for the Learning Organization , 1996 .

[15]  David P. Franz,et al.  Alternative Spreadsheet Applications: Student Learning and Attitudes , 1992 .

[16]  Wilfred W. Wu,et al.  Reliability assessment of the attitude towards computers instrument (ATCI) , 2004, Computers in Human Behavior.

[17]  Andreas Größler,et al.  Don't let history repeat itself—methodological issues concerning the use of simulators in teaching and experimentation , 2004 .

[18]  J. Wolfe.  Effective Performance Behaviors in a Simulated Policy and Decision Making Environment , 1975 .

[19]  James L. McKenney,et al.  An Evaluation of a Business Game in an MBA Curriculum , 1962 .

[20]  David Tonks,et al.  A framework for understanding learning from management simulations , 1997, J. Comput. Assist. Learn..

[21]  S. Vosniadou.  Capturing and modeling the process of conceptual change. , 1994 .

[22]  Traci Sitzmann.  Self-Reported Learning: What Are We Really Measuring? , 2007 .

[23]  M. Cowles.  N = 35: A Rule of Thumb for Psychological Researchers , 1974 .

[24]  L. Flynn,et al.  A Short, Reliable Measure of Subjective Knowledge , 1999 .

[25]  William H. Starbuck,et al.  The Effects of Advisors on Business Game Teams , 1966 .

[26]  Hugh M. Cannon,et al.  Constructs of Simulation Evaluation , 2002 .

[27]  Georges Romme,et al.  Mapping the landscape of organizational learning , 1997 .

[28]  Gerard F. Carvalho.  Evaluating Computerized Business Simulators for Objective Learning Validity , 1991 .

[29]  Kim Warren,et al.  The effective communication of system dynamics to improve insight and learning in management education , 1999, J. Oper. Res. Soc..

[30]  J. Wolfe,et al.  The Role of Management Games and Simulations in Education and Research , 1990 .

[31]  Magnus Jahre,et al.  Experimental Validation of the Learning Effect for a Pedagogical Game on Computer Fundamentals , 2009, IEEE Transactions on Education.

[32]  Steve W. J. Kozlowski,et al.  Current issues and future directions in simulation-based training in North America , 2008 .

[33]  Daniel H. Kim.  The Link between individual and organizational learning , 1997 .

[34]  John D. Sterman,et al.  Bathtub dynamics: initial results of a systems thinking inventory , 2000 .

[35]  Rex A. Thomas,et al.  Simulations: An Opportunity We Are Missing. , 1991 .

[36]  B. Brehmer,et al.  Micro-worlds and the circular relation between people and their environment , 2005 .

[37]  Erling Moxnes.  Not Only the Tragedy of the Commons: Misperceptions of Bioeconomics , 1998 .

[38]  Colin Camerer,et al.  The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework , 1999 .

[39]  David C. Lane,et al.  On a Resurgence of Management Simulations and Games , 1995 .

[40]  Kishore Sengupta,et al.  Cognitive Feedback in GDSS: Improving Control and Convergence , 1993, MIS Q..

[41]  Steven A. Cavaleri,et al.  Towards evaluation of systems‐thinking interventions: a case study , 1997 .

[42]  Dietmar Pfahl,et al.  Evaluating the learning effectiveness of using simulations in software project management education: results from a twice replicated experiment , 2004, Inf. Softw. Technol..

[43]  K. Sengupta,et al.  Alternative conceptions of feedback in dynamic decision environments: an experimental investigation , 1993 .

[44]  J.A.M. Vennix.  Mental models and computer models: design and evaluation of a computer-based learning environment for policy-making , 1990 .

[45]  L. Cohen,et al.  Research Methods in Education , 1980 .

[46]  Joseph Wolfe,et al.  The Case Approach versus Gaming in the Teaching of Business Policy: An Experimental Evaluation , 1975 .

[47]  Joseph Wolfe,et al.  A Comparative Evaluation of the Experiential Approach as a Business Policy Learning Environment , 1975 .

[48]  S. Kozlowski,et al.  Adaptive Guidance: Enhancing Self-Regulation, Knowledge, and Performance in Technology-Based Training , 2002 .

[49]  Kenneth G. Brown,et al.  Using computers to deliver training: Which employees learn and why? , 2001 .

[50]  J. Grimshaw,et al.  Constructing questionnaires based on the theory of planned behaviour: A manual for health services researchers , 2004 .

[51]  Paul A. Langley,et al.  Performance and learning in a simulation of oil industry dynamics , 2004, Eur. J. Oper. Res..

[52]  Dean Tjosvold,et al.  Book review : Group model building : facilitating team learning using system dynamics , 1998 .

[53]  James L. McKenney.  An Evaluation of a Decision Simulation as a Learning Environment , 1963 .

[54]  Joseph Wolfe,et al.  The Effects and Effectiveness of Simulations in Business Policy Teaching Applications , 1976 .

[55]  H. Stouten.  Learning from a microworld in fisheries management: The Belgian fisheries system , 2010 .

[56]  Philip H. Anderson,et al.  The Relationship between Financial Performance and Other Measures of Learning on a Simulation Exercise , 1990 .

[57]  John D. W. Morecroft,et al.  Visualising and Rehearsing Strategy , 1999 .

[58]  Francis M. Dwyer,et al.  The Effect of Online Gaming, Cognition and Feedback Type in Facilitating Delayed Achievement of Different Learning Objectives , 2005 .

[59]  Erling Moxnes,et al.  Do numerical simulation and optimization results improve management? Experimental evidence , 2003 .

[60]  L. Stolurow.  Implications of Current Research and Future Trends , 1962 .

[61]  Jerome A. Katz.  Institutionalizing Elegance: When Simulation Becomes a Requirement , 1999 .

[62]  Bent Erik Bakken.  Learning and transfer of understanding in dynamic decision environments , 1993 .

[63]  Jianwei Zhang,et al.  Supporting scientific discovery learning in a simulation environment , 2003, J. Comput. Assist. Learn..

[64]  Jochen Ludewig,et al.  Simulation in software engineering training , 2000, Proceedings of the 2000 International Conference on Software Engineering. ICSE 2000 the New Millennium.

[65]  R. Clark.  Reconsidering Research on Learning from Media , 1983 .

[66]  Goknur Kaplan Akilli,et al.  Games and Simulations: A New Approach in Education , 2009 .

[67]  Eduardo Salas,et al.  Optimizing e‐learning: Research‐based guidelines for learner‐controlled training , 2004 .

[68]  Jac A. M. Vennix,et al.  Group model building effectiveness: a review of assessment studies † , 2002 .

[69]  I. Ajzen.  The theory of planned behavior , 1991 .

[70]  Jac A. M. Vennix,et al.  Effect of system dynamics education on systems thinking inventory task performance , 2005 .

[71]  George P. Richardson,et al.  Scripts for group model building , 1997 .

[72]  Dietmar Pfahl,et al.  A CBT module with integrated simulation component for software project management education and training , 2001, J. Syst. Softw..

[73]  Marie Johnston,et al.  Application of the Theory of Planned Behaviour in Behaviour Change Interventions: A Systematic Review , 2002 .

[74]  J. Webster,et al.  Teaching Effectiveness in Technology-Mediated Distance Learning , 1997 .

[75]  Thomas C. Reeves,et al.  Pseudoscience in Computer-Based Instruction: The Case of Learner Control Research. , 1993 .

[76]  Jerry Gosen,et al.  An Exploration of Game-Derived Learning in Total Enterprise Simulations , 2001 .

[77]  Arie P. de Geus,et al.  The Living Company , 1997 .

[78]  Warren Thorngate,et al.  Why the Best Person Rarely Wins , 1987 .

[79]  Woei Hung,et al.  Enhancing systems-thinking skills with modelling , 2008, Br. J. Educ. Technol..

[80]  Kishore Sengupta,et al.  The Impact of Goals On Software Project Management: An Experimental Investigation , 1999, MIS Q..

[81]  R. A. Ellis,et al.  Learning through inquiry: student difficulties with online course-based Material , 2005, J. Comput. Assist. Learn..

[82]  P. Senge.  The fifth discipline : the art and practice of the learning organization/ Peter M. Senge , 1991 .

[83]  E. Diehl.  Effects of feedback structure on dynamic decision making , 1992 .

[84]  A. Raia.  A Study of the Educational Value of Management Games , 1966 .

[85]  William R. Dill,et al.  THE ACQUISITION OF EXPERIENCE IN A COMPLEX MANAGEMENT GAME , 1963 .

[86]  V. Smith.  Microeconomic Systems as an Experimental Science , 1982 .