Towards Assessing Students’ Fine Grained Knowledge: Using an Intelligent Tutor for Assessment

Secondary teachers across the United States are being asked to use formative assessment data to inform their classroom instruction. At the same time, critics of US government’s No Child Left Behind legislation are calling the bill “No Child Left Untested”. Among other things, critics point out that every hour spent assessing students is an hour lost from instruction. But, does it have to be? What if we better integrated assessment into classroom instruction and allowed students to learn during the test? This dissertation emphasizes using the intelligent tutoring system as an assessment system that just so happens to provide instructional assistance during the test. Usually it is believed that assessment get harder if students are allowed to learn during the test, as it’s then like trying to hit a moving target. So, my results are somewhat shocking that by providing tutoring to students while they are assessed I actually improve the assessment of students’ knowledge. Most traditional assessments treat all questions on the test as sampling a single underlying knowledge component. Yet, teachers want detailed, diagnostic reports to inform their instruction. Can we have our cake and eat it, too? In this dissertation, I provide solid evidence that a fine-grained skill model is able to

[1]  Albert T. Corbett,et al.  A Bayes Net Toolkit for Student Modeling in Intelligent Tutoring Systems , 2006, Intelligent Tutoring Systems.

[2]  Clare Lee,et al.  Assessment for Learning- putting it into practice , 2003 .

[3]  Neil T. Heffernan,et al.  Towards Live Informing and Automatic Analyzing of Student Learning: Reporting in ASSISTment System , 2007 .

[4]  Gordon I. McCalla,et al.  Granularity-Based Reasoning and Belief Revision in Student Models , 1994 .

[5]  Tiffany Barnes,et al.  The Q-matrix Method: Mining Student Response Data for Knowledge , 2005 .

[6]  Gordon I. McCalla,et al.  A Bayesian Belief Network Approach for Modeling Complex Domains , 2007 .

[7]  Kenneth R. Koedinger,et al.  An architecture for plug-in tutor agents , 1996 .

[8]  Antonija Mitrovic,et al.  Evaluating the Effects of Open Student Models on Learning , 2002, AH.

[9]  R. Stiggins From Formative Assessment to Assessment for Learning: A Path to Success in Standards-Based Schools , 2005 .

[10]  G. Bolton Reliability , 2003, Medical Humanities.

[11]  Alexander Renkl,et al.  Learning from Worked-Out-Examples: A Study on Individual Differences , 1997, Cogn. Sci..

[12]  Robert J. Jannarone,et al.  Conjunctive item response theory kernels , 1986 .

[13]  Brian W. Junker,et al.  Using On-line Tutoring Records to Predict End-of-Year Exam Scores : Experience with the ASSISTments Project and MCAS 8 th Grade Mathematics , 2006 .

[14]  Willem J. van der Linden,et al.  Using Response Times for Item Selection in Adaptive Testing , 2008 .

[15]  R. Sternberg Human Abilities: An Information Processing Approach , 1985 .

[16]  Albert T. Corbett,et al.  Does Help Help? Introducing the Bayesian Evaluation and Assessment Methodology , 2008, Intelligent Tutoring Systems.

[17]  Neil T. Heffernan,et al.  The eXtensible Tutor Architecture: A New Foundation for ITS , 2005, AIED.

[18]  R. J. Mokken,et al.  Handbook of modern item response theory , 1997 .

[19]  Beverly Park Woolf,et al.  Transfer Learning and Representation Discovery in Intelligent Tutoring Systems , 2009, AIED.

[20]  Zachary A. Pardos,et al.  Determining the Significance of Item Order In Randomized Problem Sets , 2009, EDM.

[21]  Cristina Conati,et al.  On-Line Student Modeling for Coached Problem Solving Using Bayesian Networks , 1997 .

[22]  Neil T. Heffernan,et al.  Detection and Analysis of Off-Task Gaming Behavior in Intelligent Tutoring Systems , 2006, Intelligent Tutoring Systems.

[23]  Neil T. Heffernan,et al.  Scaffolding vs. Hints in the Assistment System , 2006, Intelligent Tutoring Systems.

[24]  J. Patvarczki,et al.  Scalability and Robustness in the Domain of Web-Based Tutoring , 2009 .

[25]  Peter Brusilovsky,et al.  Open user profiles for adaptive news systems: help or harm? , 2007, WWW '07.

[26]  Peter Brusilovsky,et al.  User Modeling in a Distributed E-Learning Architecture , 2005, User Modeling.

[27]  Brian W. Junker,et al.  Predicting end-of-year accountability assessment scores from monthly student records in an online tutoring system , 2006 .

[28]  Jim E. Greer,et al.  Adaptive Assessment Using Granularity Hierarchies and Bayesian Nets , 1996, Intelligent Tutoring Systems.

[29]  Kenneth R. Koedinger,et al.  Distinguishing Qualitatively Different Kinds of Learning Using Log Files and Learning Curves , 2005 .

[30]  Neil T. Heffernan,et al.  Looking for Sources of Error in Predicting Student's Knowledge , 2005 .

[31]  Vincent Aleven,et al.  An effective metacognitive strategy: learning by doing and explaining with a computer-based Cognitive Tutor , 2002, Cogn. Sci..

[32]  Fritz Drasgow,et al.  Recovery of Two- and Three-Parameter Logistic Item Characteristic Curves: A Monte Carlo Study , 1982 .

[33]  Murali Mani,et al.  Using Mixed-Effects Modeling to Compare Different Grain-Sized Skill Models , 2006 .

[34]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[35]  Allen and Rosenbloom Paul S. Newell,et al.  Mechanisms of Skill Acquisition and the Law of Practice , 1993 .

[36]  Barbara Means,et al.  Implementing Data-Informed Decision Making in Schools: Teacher Access, Supports and Use. , 2009 .

[37]  Lynn Olson State Test Programs Mushroom as NCLB Mandate Kicks in: Nearly Half of States Are Expanding Their Testing Programs to Additional Grades This School Year to Comply with the Federal No Child Left Behind Act. , 2005 .

[38]  Peter Brusilovsky,et al.  User Models for Adaptive Hypermedia and Adaptive Educational Systems , 2007, The Adaptive Web.

[39]  Ricardo Conejo,et al.  Introducing Prerequisite Relations in a Multi-layered Bayesian Student Model , 2005, User Modeling.

[40]  B. Junker,et al.  Do Skills Combine Additively to Predict Task Di culty in Eighth-grade Mathematics ? , 2006 .

[41]  Kenneth R. Koedinger,et al.  Performance Factors Analysis - A New Alternative to Knowledge Tracing , 2009, AIED.

[42]  Susan E. Embretson,et al.  A multidimensional latent trait model for measuring learning and change , 1991 .

[43]  A. Raftery Bayesian Model Selection in Social Research , 1995 .

[44]  Mark J. Gierl,et al.  The Attribute Hierarchy Method for Cognitive Assessment: A Variation on Tatsuoka's Rule-Space Approach , 2004 .

[45]  Sandra Katz,et al.  Modeling the Student in Sherlock II , 1992 .

[46]  E. Muraki,et al.  Full-Information Item Factor Analysis , 1988 .

[47]  Neil T. Heffernan,et al.  Addressing the assessment challenge with an online system that tutors as it assesses , 2009, User Modeling and User-Adapted Interaction.

[48]  Robert J. Mislevy,et al.  The role of probability-based inference in an intelligent tutoring system , 2005, User Modeling and User-Adapted Interaction.

[49]  Neil T. Heffernan,et al.  Why Are Algebra Word Problems Difficult? Using Tutorial Log Files and the Power Law of Learning to Select the Best Fitting Cognitive Model , 2004, Intelligent Tutoring Systems.

[50]  Donald Hedeker,et al.  Longitudinal Data Analysis , 2006 .

[51]  Beverly Park Woolf,et al.  Improving Intelligent Tutoring Systems: Using Expectation Maximization to Learn Student Skill Levels , 2006, Intelligent Tutoring Systems.

[52]  Neil T. Heffernan,et al.  Using Learning Decomposition to Analyze Instructional Effectiveness in the ASSISTment System , 2009, AIED.

[53]  Neil T. Heffernan,et al.  The Assistment Builder: A Rapid Development Tool for ITS , 2005, AIED.

[54]  Neil T. Heffernan,et al.  Predicting State Test Scores Better with Intelligent Tutoring Systems: Developing Metrics to Measure Assistance Required , 2006, Intelligent Tutoring Systems.

[55]  K. Tatsuoka RULE SPACE: AN APPROACH FOR DEALING WITH MISCONCEPTIONS BASED ON ITEM RESPONSE THEORY , 1983 .

[56]  P. Brusilovsky,et al.  NewsMe: A Case Study for Adaptive News Systems with Open User Model , 2007, Third International Conference on Autonomic and Autonomous Systems (ICAS'07).

[57]  John R. Anderson,et al.  Student modeling in the ACT Programming Tutor. , 1995 .

[58]  Judy Kay,et al.  Student Models that Invite the Learner In: The SMILI: () Open Learner Modelling Framework , 2007, Int. J. Artif. Intell. Educ..

[59]  Russell G. Almond,et al.  On the Structure of Educational Assessments, CSE Technical Report. , 2003 .

[60]  C. Boston The Concept of Formative Assessment. , 2002 .

[61]  Susan Bull,et al.  An Intelligent Learning Environment with an Open Learner Model for the Desktop PC and Pocket PC , 2003 .

[62]  Kenneth R. Koedinger,et al.  An Investigation of Teachers' Beliefs of Students' Algebra Development , 2000, Cognition and Instruction.

[63]  R. C. Murray A Decision-Theoretic Approach for Selecting Tutorial Discourse Actions , 2001 .

[64]  Judy Kay,et al.  Learner Know Thyself: Student Models to Give Learner Control and Responsibility , 1997 .

[65]  Neil T. Heffernan,et al.  Lessons Learned from Scaling Up a Web-Based Intelligent Tutoring System , 2008, Intelligent Tutoring Systems.

[66]  Robert J. Sternberg,et al.  Dynamic Testing: The Nature and Measurement of Learning Potential , 2001 .

[67]  Ronald J. M. M. Does,et al.  A Distribution-Free Approach for Comparing Growth of Knowledge , 1994 .

[68]  Kurt VanLehn,et al.  The Andes Physics Tutoring System: Lessons Learned , 2005, Int. J. Artif. Intell. Educ..

[69]  Kenneth R. Koedinger,et al.  Learning Factors Analysis - A General Method for Cognitive Model Evaluation and Improvement , 2006, Intelligent Tutoring Systems.

[70]  Neil T. Heffernan,et al.  What Level of Tutor Interaction is Best? , 2007, AIED.

[71]  Neil T. Heffernan,et al.  Informing Teachers Live about Student Learning: Reporting in Assistment System , 2005 .

[72]  Neil T. Heffernan,et al.  The ASSISTment Builder: Towards an Analysis of Cost Effectiveness of ITS Creation , 2006, FLAIRS Conference.

[73]  Neil T. Heffernan,et al.  The AS SISTment Builder: Supporting the Life Cycle of ITS Content Creation , 2008 .

[74]  Mark J. Gierl,et al.  Cognitive diagnostic assessment for education: Theory and applications. , 2007 .

[75]  Ryan Shaun Joazeiro de Baker,et al.  Detecting Student Misuse of Intelligent Tutoring Systems , 2004, Intelligent Tutoring Systems.

[76]  Mark J. Gierl,et al.  Using the Attribute Hierarchy Method to Make Diagnostic Inferences about Examinees' Cognitive Skills in Algebra on the SAT. , 2008 .

[77]  C. Lebiere,et al.  The Atomic Components of Thought , 1998 .

[78]  Neil T. Heffernan,et al.  Using Learning Decomposition and Bootstrapping with Randomization to Compare the Impact of Different Educational Interventions on Learning , 2009, EDM.

[79]  Neil T. Heffernan,et al.  Addressing the testing challenge with a web-based e-assessment system that tutors as it assesses , 2006, WWW '06.

[80]  J. Beck,et al.  Comparing Pedagogical Approaches for Teaching the Control of Variables Strategy , 2009 .

[81]  P. Black,et al.  Inside the Black Box: Raising Standards through Classroom Assessment , 2010 .

[82]  R. Charles Murray,et al.  Reducing the Knowledge Tracing Space , 2009, EDM.

[83]  B. Junker,et al.  Cognitive Assessment Models with Few Assumptions, and Connections with Nonparametric Item Response Theory , 2001 .

[84]  Georg Rasch,et al.  Probabilistic Models for Some Intelligence and Attainment Tests , 1981, The SAGE Encyclopedia of Research Design.

[85]  Zachary A. Pardos,et al.  Using Fine-Grained Skill Models to Fit Student Performance with , 2006 .

[86]  van der Linden,et al.  A hierarchical framework for modeling speed and accuracy on test items , 2007 .

[87]  Raymond J. Mooney,et al.  A Novel Application of Theory Refinement to Student Modeling , 1996, AAAI/IAAI, Vol. 1.

[88]  Joseph E. Beck,et al.  Using learning decomposition to analyze student fluency development , 2005 .

[89]  Jack Mostow,et al.  Evaluating tutors that listen: an overview of project LISTEN , 2001 .

[90]  J WIM,et al.  A HIERARCHICAL FRAMEWORK FOR MODELING SPEED AND ACCURACY ON TEST ITEMS , 2007 .

[91]  Thomas L. Dean,et al.  A Model for Projection and Action , 1989, IJCAI.

[92]  Jack Mostow,et al.  Automatically Assessing Oral Reading Fluency in a Computer Tutor that Listens , 2004 .

[93]  Joseph E. Beck,et al.  Back to the future: a non-automated method for constructing transfer models , 2009, EDM.

[94]  Tiffany Barnes Evaluation of the q-matrix Method in Understanding Student Logic Proofs , 2006, FLAIRS Conference.

[95]  Neil T. Heffernan,et al.  Can we predict which groups of questions students will learn from? , 2008, EDM.

[96]  John R. Anderson,et al.  Cognitive Tutors: Lessons Learned , 1995 .

[97]  J. Ware,et al.  Applied Longitudinal Analysis , 2004 .

[98]  Ryan Shaun Joazeiro de Baker,et al.  Do Performance Goals Lead Students to Game the System? , 2005, AIED.

[99]  John R. Anderson,et al.  Rules of the Mind , 1993 .

[100]  Albert T. Corbett,et al.  Student Modeling in the ACT Programming Tutor: Adjusting a Procedural Learning Model With Declarative Knowledge , 1997 .

[101]  Susan Bull,et al.  Use and Trust of Simple Independent Open Learner Models to Support Learning within and across Courses , 2009, UMAP.

[102]  K. Tatsuoka Toward an Integration of Item-Response Theory and Cognitive Error Diagnosis. , 1987 .

[103]  T. Schack,et al.  Dynamic testing , 2003 .

[104]  K. Koedinger,et al.  Cyberinfrastructure for Education and Learning for the Future: a vision and research agenda , 2005 .

[105]  Leena M. Razzaq,et al.  Developing Fine-Grained Transfer Models in the ASSISTment System , 2007 .

[106]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[107]  Neil T. Heffernan,et al.  Can an Intelligent Tutoring System Predict Math Proficiency as Well as a Standarized Test? , 2008, EDM.

[108]  Joseph E. Beck,et al.  Using Knowledge Tracing in a Noisy Environment to Measure Student Reading Proficiencies , 2006, Int. J. Artif. Intell. Educ..

[109]  Dylan Wiliam,et al.  Formative Assessment: Getting the Focus Right , 2006 .

[110]  Vlado Glavinić,et al.  Intelligent Tutoring Systems in E-Learning Environments: Design, Implementation and Evaluation , 2010 .

[111]  K. Koedinger,et al.  Automating Cognitive Model Improvement by A * Search and Logistic Regression , 2005 .

[112]  Russell G. Almond,et al.  Bayesian networks: A teacher's view , 2009, Int. J. Approx. Reason..

[113]  William J. Clancey,et al.  Knowledge-based tutoring: the GUIDON program , 1987 .

[114]  S. Graf,et al.  Adaptive and Intelligent Web-Based Educational Systems , 2009 .

[115]  Neil T. Heffernan,et al.  Opening the Door to Non-programmers: Authoring Intelligent Tutor Behavior by Demonstration , 2004, Intelligent Tutoring Systems.

[116]  Kenneth R. Koedinger,et al.  Learning Factors Transfer Analysis: Using Learning Curve Analysis to Automatically Generate Domain Models , 2009, EDM.

[117]  James M. Leszczenski,et al.  What’s in a Word? Extending Learning Factors Analysis to Model Reading Transfer , 2007 .

[118]  Zachary A. Pardos,et al.  Detecting the Learning Value of Items In a Randomized Problem Set , 2009, AIED.

[119]  Zachary A. Pardos,et al.  Analyzing Fine-Grained Skill Models Using Bayesian and Mixed Effects Methods , 2007, AIED.

[120]  Susan Bull,et al.  Helping Learners to Understand Themselves with a Learner Model Open to Students, Peers and Instructors , 2002 .

[121]  Anthony E. Kelly,et al.  DIAGNOSING KNOWLEDGE STATES IN ALGEBRA USING THE RULE SPACE MODEL , 1992 .

[122]  Paul Brna,et al.  Applying Interactive Open Learner Models to Learning Technical Terminology , 2001, User Modeling.