Lessons Learned and Best Practices of Stealth Assessment

Stealth assessment provides an innovative way to assess and ultimately support knowledge, skills, and other personal attributes within learning and gaming environments without disrupting students' flow. In this paper, the authors briefly discuss two challenges they encountered during the development of stealth assessments in two past projects i.e., utility issues related to log files and validation issues related to in-game measures. They also present successful examples of designing and testing stealth assessments and describe the steps they are taking to apply the lessons they have learned to the ongoing development of a stealth assessment for problem solving skills. The authors conclude with suggestions for future research.

[1]  Alexandra Rankin Macgill,et al.  Teens, Video Games, and Civics: Teens' Gaming Experiences Are Diverse and Include Significant Social Interaction and Civic Engagement. , 2008 .

[2]  V. Shute,et al.  Stealth Assessment: Measuring and Supporting Learning in Video Games , 2013 .

[3]  D. Paulhus.  Measurement and control of response bias. , 1991 .

[4]  Ryan S. Baker,et al.  The State of Educational Data Mining in 2009: A Review and Future Visions. , 2009, EDM 2009.

[5]  Yoon Jeon Kim,et al.  Does Playing the World of Goo Facilitate Learning? , 2010 .

[6]  Russell G. Almond,et al.  How Task Features Impact Evidence From Assessments Embedded in Simulations and Games , 2014, Measurement : interdisciplinary research and perspectives.

[7]  V. Shute,et al.  Assessment and Learning of Qualitative Physics in Newton's Playground , 2013, AIED.

[8]  V. Shute.  SteAlth ASSeSSment in computer-BASed GAmeS to Support leArninG , 2011 .

[9]  N. Schmitt.  Method bias: The importance of theory and measurement , 1994 .

[10]  A. Caspi,et al.  The Power of Personality: The Comparative Validity of Personality Traits, Socioeconomic Status, and Cognitive Ability for Predicting Important Life Outcomes , 2007, Perspectives on psychological science : a journal of the Association for Psychological Science.

[11]  Russell G. Almond,et al.  Debugging the Evidence Chain , 2013, UAI Application Workshops.

[12]  Valerie J. Shute,et al.  The validity of a game-based assessment of persistence , 2013, Comput. Hum. Behav..

[13]  Richard I. Lanyon,et al.  Personality assessment, 3rd ed. , 1997 .

[14]  Tim Wright,et al.  An investigation of the validity of the virtual spatial navigation assessment , 2013, Front. Psychol..

[15]  D. Simons,et al.  The effects of video game playing on attention, memory, and executive control. , 2008, Acta psychologica.

[16]  Robert W. Weisberg,et al.  An examination of the alleged role of "fixation" in the solution of several "insight" problems. , 1981 .

[17]  J. Raven.  STANDARDIZATION OF PROGRESSIVE MATRICES, 1938 , 1941 .

[18]  B. Hommel,et al.  Development and Validity of a Dutch Version of the Remote Associates Task: An Item-Response Theory Approach. , 2012 .

[19]  Valerie J. Shute,et al.  Advances in the Science of Assessment , 2016 .

[20]  Richard Van Eck,et al.  Aligning Problem Solving and Gameplay : A Model for Future Research and Design , 2010 .

[21]  S. Mednick.  The associative basis of the creative process. , 1962, Psychological review.

[22]  Russell G. Almond,et al.  Models for Conditional Probability Tables in Educational Assessment , 2001, AISTATS.

[23]  Michel C. Desmarais,et al.  A Bayesian Student Model without Hidden Nodes and its Comparison with Item Response Theory , 2005, Int. J. Artif. Intell. Educ..

[24]  James N. MacGregor,et al.  Human Performance on Insight Problem Solving: A Review , 2011, J. Probl. Solving.

[25]  Valerie J. Shute,et al.  Conceptual Framework for Modeling , Assessing and Supporting Competencies within Game Environments , 2010 .

[26]  Weinan Zhao,et al.  The relationship between video game use and a performance-based measure of persistence , 2013, Comput. Educ..

[27]  Matthew Ventura,et al.  C HAPTER 8 ‒ Assessing Persistence in Educational Games , 2014 .

[28]  Russell G. Almond,et al.  Using Evidence Centered Design to Think About Assessments , 2010 .

[29]  Valerie J. Shute,et al.  The power of play: The effects of Portal 2 and Lumosity on cognitive and noncognitive skills , 2015, Comput. Educ..

[30]  R. Almond,et al.  Focus Article: On the Structure of Educational Assessments , 2003 .

[31]  A. Poropat.  A meta-analysis of the five-factor model of personality and academic performance. , 2009, Psychological bulletin.

[32]  S. Wilson.  What Video Games Have to Teach Us about Learning and Literacy , 2006 .