Interpreting Evidence-of-Learning: Educational research in the era of big data

Abstract In this article, we argue that big data can offer new opportunities and roles for educational researchers. In the traditional model of evidence-gathering and interpretation in education, researchers are independent observers, who pre-emptively create instruments of measurement, and insert these into the educational process in specialized times and places (a pre-test or post-test, a survey, an interview, a focus group). The ‘big data’ approach is to collect data through practice-integrated research. If a record is kept of everything that happens, then it is possible analyze what happened, ex post facto. Data collection is embedded. It is on-the-fly and ever-present. With the relevant analysis and presentation software, the data is readable in the form of data reports, analytics dashboards and visualizations. We explore the methodological consequences of these developments for research methods.

[1]  Mary Kalantzis,et al.  Towards a Semantic web: Connecting Knowledge in Academic Research, Bill Cope, Mary Kalantzis, Liam Magee. Chandos Publishing, Oxford (2011), 525 p. $110.00, ISBN: 978-1-84334-601-2 , 2011 .

[2]  Tim O'Reilly,et al.  What is Web 2.0: Design Patterns and Business Models for the Next Generation of Software , 2007 .

[3]  Charles Anderson,et al.  The end of theory: The data deluge makes the scientific method obsolete , 2008 .

[4]  Judy Kay,et al.  Learner Control , 2001, User Modeling and User-Adapted Interaction.

[5]  Cognitive Board on Behavioral Proposed Revisions to the Common Rule for the Protection of Human Subjects in the Behavioral and Social Sciences , 2014 .

[6]  Zoran Popovic,et al.  Learning Individual Behavior in an Educational Game: A Data-Driven Approach , 2014, EDM.

[7]  Norbert Wiener,et al.  Cybernetics, Second Edition: or the Control and Communication in the Animal and the Machine , 1965 .

[8]  Zoran Zdravev,et al.  Big data for education data mining, data analytics and web dashboards , 2015 .

[9]  Edmund A. Mennis The Wisdom of Crowds: Why the Many Are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations , 2006 .

[10]  Arthur C. Graesser,et al.  Coh-Metrix: An automated tool for theoretical and applied natural language processing , 2011 .

[11]  R. Guimerà,et al.  Complex Systems View of Educational Policy Research , 2010, Science.

[12]  Alexis Wichowski Survival of the Fittest Tag: Folksonomies, Findability, and the Evolution of Information Organization , 2009, First Monday.

[13]  Elliot Stern,et al.  DFID Working Paper 38. Broadening the range of designs and methods for impact evaluations. , 2012 .

[14]  Eric Gossett,et al.  Big Data: A Revolution That Will Transform How We Live, Work, and Think , 2015 .

[15]  Mary Kalantzis,et al.  New Spaces and Old Places: An Analysis of Writing Assessment Software , 2011 .

[16]  Donna Charlevoix,et al.  Do professors matter?: using an a/b test to evaluate the impact of instructor involvement on MOOC student outcomes , 2014, L@S.

[17]  Ryan S. Baker,et al.  The Potentials of Educational Data Mining for Researching Metacognition, Motivation and Self-Regulated Learning , 2013, EDM 2013.

[18]  Ryan Shaun Joazeiro de Baker,et al.  Educational Data Mining and Learning Analytics: Applications to Constructionist Research , 2014, Technology, Knowledge and Learning.

[19]  K. Gergen,et al.  Epistemology and Measurement : Paradigms and Practices , 2013 .

[20]  J. I The Design of Experiments , 1936, Nature.

[21]  Justin Reich,et al.  Privacy, Anonymity, and Big Data in the Social Sciences , 2014 .

[22]  Beverly Park Woolf,et al.  A Roadmap for Education Technology , 2011 .

[23]  Robert J. Mislevy,et al.  Design and Discovery in Educational Assessment: Evidence-Centered Design, Psychometrics, and Educational Data Mining , 2012, EDM 2012.

[24]  Justin Reich,et al.  Privacy, anonymity, and big data in the social sciences , 2014, Commun. ACM.

[25]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[26]  Ryan Shaun Joazeiro de Baker,et al.  New Potentials for Data-Driven Intelligent Tutoring System Development and Optimization , 2013, AI Mag..

[27]  Ann L. Brown,et al.  Training in Self-Explanation and Self-Regulation Strategies: Investigating the Effects of Knowledge Acquisition Activities on Problem Solving , 1995 .

[28]  Jeffrey T. Hancock,et al.  Experimental evidence of massive-scale emotional contagion through social networks , 2014, Proceedings of the National Academy of Sciences.

[29]  Allen H. Renear,et al.  Strategic Reading, Ontologies, and the Future of Scientific Publishing , 2009, Science.

[30]  Brian C. Nelson,et al.  Evidence-centered Design for Diagnostic Assessment within Digital Learning Environments: Integrating Modern Psychometrics and Educational Data Mining , 2012, EDM 2012.

[31]  Mary Kalantzis,et al.  New Learning: Elements of a Science of Education , 2008 .

[32]  Uwe Hansmann,et al.  Agile Software Development: Best Practices for Large Software Development Projects , 2009 .

[33]  Simon Buckingham Shum,et al.  Epistemology, pedagogy, assessment and learning analytics , 2013, LAK '13.

[34]  H. Chad Lane,et al.  AI Grand Challenges for Education , 2013, AI Mag..

[35]  H. Chad Lane,et al.  Intelligent Learning Technologies: Applications of Artificial Intelligence to Contemporary and Emerging Educational Challenges , 2013, AI Mag..

[36]  Robert C. Martin Agile Software Development, Principles, Patterns, and Practices , 2002 .

[37]  Hod Lipson,et al.  Data smashing: uncovering lurking order in data , 2014, Journal of The Royal Society Interface.

[38]  Ann L. Brown,et al.  How people learn: Brain, mind, experience, and school. , 1999 .

[39]  Dylan Wiliam,et al.  Embedded Formative Assessment , 2011 .

[40]  The Microgenetic Method: Current Work and Extensions to Classroom Research , 2006 .

[41]  Jan-Willem Strijbos,et al.  Unravelling peer assessment: Methodological, functional, and conceptual developments , 2010 .

[42]  R. Glaser,et al.  Knowing What Students Know: The Science and Design of Educational Assessment , 2001 .

[43]  Tiffany Barnes,et al.  Toward Automatic Hint Generation for Logic Proof Tutoring Using Historical Student Data , 2008, Intelligent Tutoring Systems.

[44]  Anthony J. G. Hey,et al.  The Fourth Paradigm: Data-Intensive Scientific Discovery [Point of View] , 2011 .

[45]  Patrick Carmichael,et al.  Case-Based Learning, Pedagogical Innovation, and Semantic Web Technologies , 2012, IEEE Transactions on Learning Technologies.

[46]  Tony Hey,et al.  The Fourth Paradigm: Data-Intensive Scientific Discovery , 2009 .

[47]  Mary Kalantzis,et al.  Changing knowledge ecologies and the transformation of the scholarly journal , 2014 .

[48]  Pramod K. Varshney,et al.  Reliable classification by unreliable crowds , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[49]  Viktor Mayer-Schnberger,et al.  Big Data: A Revolution That Will Transform How We Live, Work, and Think , 2013 .

[50]  J. Pearl Causality: Models, Reasoning and Inference , 2000 .

[51]  Lawrence Lessig,et al.  Remix: Making Art and Commerce Thrive in the Hybrid Economy , 2008 .

[52]  C. Lynch Big data: How do your data grow? , 2008, Nature.

[53]  Kenneth R. Koedinger,et al.  A Data Repository for the EDM Community: The PSLC DataShop , 2010 .

[54]  Courtney B. Cazden,et al.  Classroom Discourse: The Language of Teaching and Learning. Second Edition. , 2001 .

[55]  Cristina Conati,et al.  Student Modeling: Supporting Personalized Instruction, from Problem Solving to Exploratory Open Ended Activities , 2013, AI Mag..

[56]  Gregory K. W. K. Chung,et al.  Toward the Relational Management of Educational Measurement Data , 2014, Teachers College Record: The Voice of Scholarship in Education.

[57]  David M. Nichols,et al.  Metadatapedia: a proposal for aggregating metadata on data archiving , 2012, iConference '12.

[58]  George Siemens,et al.  Learning analytics and educational data mining: towards communication and collaboration , 2012, LAK.

[59]  Ryan S. Baker,et al.  Educational Data Mining and Learning Analytics , 2014 .

[60]  Danielle S. McNamara,et al.  Who's in Control?: Categorizing Nuanced Patterns of Behaviors within a Game-Based Intelligent Tutoring System , 2014, EDM.

[61]  F. H. Adler Cybernetics, or Control and Communication in the Animal and the Machine. , 1949 .

[62]  Àngela Nebot,et al.  Applying Data Mining Techniques to e-Learning Problems , 2007 .

[63]  Kristen E. DiCerbo,et al.  Impacts of the Digital Ocean on Education , 2014 .

[64]  Michael C. Mozer,et al.  Integrating latent-factor and knowledge-tracing models to predict individual differences in learning , 2014, EDM.

[65]  Barbara Schneider Estimating causal effects : using experimental and observational designs : a think tank white paper , 2007 .

[66]  Gene V. Glass,et al.  Meta-Analysis: The Quantitative Synthesis of Research Findings , 2006 .

[67]  John Hattie,et al.  Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement , 2008 .

[68]  Philip H. Winne,et al.  How Software Technologies Can Improve Research on Learning and Bolster School Reform , 2006 .

[69]  Silvia Micheletta John Hattie, Visible Learning. A synthesis of over 800 meta-analyses relating to achievement. London & New York: Routledge (2009). John Hattie, Visible Learning for teachers. Maximizing impact on learning. London & New York: Routledge (2012). , 2013 .

[70]  Zhenghao Chen,et al.  Tuned Models of Peer Assessment in MOOCs , 2013, EDM.

[71]  Elizabeth Bagley,et al.  Science in Writing: Learning Scientific Argument in Principle and Practice , 2013 .

[72]  R. Almond,et al.  Making Sense of Data From Complex Assessments , 2002 .

[73]  Hua-Hua Chang,et al.  Psychometrics Behind Computerized Adaptive Testing , 2015, Psychometrika.

[74]  Jennifer Heath,et al.  Contemporary Privacy Theory Contributions to Learning Analytics , 2014, J. Learn. Anal..

[75]  Marie Bienkowski,et al.  Enhancing Teaching and Learning Through Educational Data Mining and Learning Analytics: An Issue Brief , 2012 .

[76]  Sarah L. Shreeves The Role of Repositories in the Future of the Journal , 2014 .

[77]  Carlos Monroy,et al.  STEMscopes: contextualizing learning analytics in a K-12 science curriculum , 2013, LAK '13.

[78]  B. Cope,et al.  New Pedagogies of Motivation: Reconstructing and Repositioning Motivational Constructs in the Design of Learning Technologies , 2013 .

[79]  J. Greeno THE SITUATIVITY OF KNOWING, LEARNING, AND RESEARCH , 1998 .

[80]  Judy Kay,et al.  Empowering Teachers to Design Learning Resources with Metacognitive Interface Elements , 2013 .

[81]  Simone Sacchi,et al.  Identifying content and levels of representation in scientific data , 2012, ASIST.