Challenges for IT-Enabled Formative Assessment of Complex 21st Century Skills

In this article, we identify and examine opportunities for formative assessment provided by information technologies (IT) and the challenges which these opportunities present. We address some of these challenges by examining key aspects of assessment processes that can be facilitated by IT: datafication of learning; feedback and scaffolding; peer assessment and peer feedback. We then consider how these processes may be applied in relation to the assessment of horizontal, general complex 21st century skills (21st CS), which are still proving challenging to incorporate into curricula as well as to assess. 21st CS such as creativity, complex problem solving, communication, collaboration and self-regulated learning contain complex constructs incorporating motivational and affective components. Our analysis has enabled us to make recommendations for policy, practice and further research. While there is currently much interest in and some progress towards the development of learning/assessment analytics for assessing 21st CS, the complexity of assessing such skills, together with the need to include affective aspects means that using IT-enabled techniques will need to be combined with more traditional methods of teacher assessment as well as peer assessment for some time to come. Therefore learners, teachers and school leaders must learn how to manage the greater variety of sorts and sources of feedback including resolving tensions of inconsistent feedback from different sources.

[1]  L. Vygotsky,et al.  Thought and Language , 1963 .

[2]  J. Dewey,et al.  Interest and Effort in Education , 1975 .

[3]  P. Black,et al.  Assessment and Classroom Learning , 1998 .

[4]  K. Topping Peer Assessment Between Students in Colleges and Universities , 1998 .

[5]  P. Black,et al.  Developing the theory of formative assessment , 2009 .

[6]  D. Carless,et al.  Peer feedback: the learning element of peer assessment , 2006 .

[7]  John Hattie,et al.  Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement , 2008 .

[8]  D. Sadler Beyond feedback: developing student capability in complex appraisal , 2010 .

[9]  S. Mann,et al.  How Do We Measure Affective Learning in Higher Education? , 2011 .

[10]  Trond Eiliv Hauge,et al.  Blogs and the design of reflective peer‐to‐peer technology‐enhanced learning and formative assessment , 2011 .

[11]  Dirk T. Tempelaar,et al.  Formative assessment and learning analytics , 2013, LAK '13.

[12]  Chris Dede,et al.  Challenges to learning and schooling in the digital networked world of the 21st century , 2013, J. Comput. Assist. Learn..

[13]  Cath Ellis,et al.  Broadening the scope and increasing the usefulness of learning analytics: The case for assessment analytics , 2013, Br. J. Educ. Technol..

[14]  Mee-Chin Wee,et al.  An Analysis of Web-Based Formative Assessment Systems Used in E-Learning Environment , 2013, 2013 IEEE 13th International Conference on Advanced Learning Technologies.

[15]  Mary Webb,et al.  Challenges for information technology supporting educational assessment , 2013, J. Comput. Assist. Learn..

[16]  Yuan-Cheng Lai,et al.  Online formative assessments with social network awareness , 2013, Comput. Educ..

[17]  Erik Duval,et al.  Learning dashboards: an overview and future research opportunities , 2013, Personal and Ubiquitous Computing.

[18]  Erik Duval,et al.  Learning Analytics Dashboard Applications , 2013 .

[19]  Natalie Förster,et al.  Learning progress assessment and goal setting: Effects on reading achievement, reading motivation and reading self-concept , 2014 .

[20]  Paulo Oliveira,et al.  A system for formative assessment and monitoring of students' progress , 2014, Comput. Educ..

[21]  David Carless,et al.  Developing Assessment for Productive Learning in Confucian-Influenced Settings , 2014 .

[22]  Dirk Ifenthaler Toward automated computer-based visualization and assessment of team-based performance , 2014 .

[23]  Ching-Huei Chen An adaptive scaffolding e-learning system for middle school students’ physics learning , 2014 .

[24]  P. Black Formative assessment – an optimistic but incomplete vision , 2015 .

[25]  Francisco J. García-Peñalvo,et al.  Using Learning Analytics to improve teamwork assessment , 2015, Comput. Hum. Behav..

[26]  James C. Lester,et al.  Face-to-Face Interaction with Pedagogical Agents, Twenty Years Later , 2016, International Journal of Artificial Intelligence in Education.

[27]  Patrick Griffin,et al.  Assessment and teaching of 21st century skills : methods and approach , 2015 .

[28]  Mary Webb,et al.  Data science in educational assessment , 2015, Education and Information Technologies.

[29]  Matthew Berland,et al.  AMOEBA: Designing for collaboration in computer science classrooms through live learning analytics , 2015, International Journal of Computer-Supported Collaborative Learning.

[30]  J. Foley,et al.  Developing Effective Affective Assessment Practices , 2015 .

[31]  Mary Webb,et al.  Technology enhanced assessment in complex collaborative settings , 2015, Education and Information Technologies.

[32]  Samuel Greiff,et al.  Assessment of Complex Problem Solving: What We Know and What We Don’t Know , 2016 .

[33]  M. Sanchez-Ruiz,et al.  Non-cognitive skills and factors in educational attainment , 2016 .

[34]  P. Griffin,et al.  Assessment of Collaborative Problem Solving in Education Environments , 2016 .

[35]  Samuel Greiff,et al.  Understanding students' performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files , 2016, Comput. Hum. Behav..

[36]  Il-Hyun Jo,et al.  Effects of learning analytics dashboard: analyzing the relations among dashboard utilization, satisfaction, and learning achievement , 2015, Asia Pacific Education Review.

[37]  Jason M. Harley,et al.  Developing Emotion-Aware, Advanced Learning Technologies: A Taxonomy of Approaches and Features , 2016, International Journal of Artificial Intelligence in Education.

[38]  B. Lucas A Five-Dimensional Model of Creativity and its Assessment in Schools , 2016 .

[39]  Uwe Maier,et al.  Effects of a computer-assisted formative assessment intervention based on multiple-tier diagnostic items and different feedback types , 2016, Comput. Educ..

[40]  Samuel Greiff,et al.  Measuring problem solving skills via stealth assessment in an engaging video game , 2016, Comput. Hum. Behav..

[41]  J. Michael Spector,et al.  Technology Enhanced Formative Assessment for 21st Century Learning , 2016, J. Educ. Technol. Soc..

[42]  Anna Wilson,et al.  Assessing the unassessable: making learning visible in undergraduates’ experiences of scientific research , 2016 .

[43]  K. Geisinger 21st Century Skills: What Are They and How Do We Assess Them? , 2016 .

[44]  J. Fraillon,et al.  Conceptualizing and Measuring Computer and Information Literacy in Cross-National Contexts , 2016 .

[45]  Philip H. Winne,et al.  Generating Learning Analytics to Improve Learners' Metacognitive Skills Using nStudy Trace Data and the ICAP Framework , 2016, LAL@LAK.

[46]  K. Ercikan,et al.  In Search of Validity Evidence in Support of the Interpretation and Use of Assessments of Complex Constructs: Discussion of Research on Assessing 21st Century Skills , 2016 .

[47]  V.J. Shute,et al.  Review of computer-based assessment for learning in elementary and secondary education , 2017, J. Comput. Assist. Learn..

[48]  Matthias Nückles,et al.  Formative computer-based feedback in the university classroom: Specific concept maps scaffold students' writing , 2017, Comput. Hum. Behav..

[49]  Kati Mäkitalo-Siegl,et al.  TPACK updated to measure pre-service teachers’ twenty-first century skills , 2017 .

[50]  Duygu Bektik,et al.  Learning Analytics for Academic Writing through Automatic Identification of Meta-discourse , 2017 .

[51]  Andreas Lachner,et al.  Formative computer-based feedback in the university classroom: Specific concept maps scaffold students' writing , 2017 .

[52]  Eva Dobozy,et al.  The synergistic and dynamic relationship between learning design and learning analytics , 2017 .

[53]  Amary Mey,et al.  Development of a phenomenologically derived method to assess affective learning in student journals following impactive educational experiences , 2017, Medical teacher.

[54]  Fazilat Siddiq,et al.  Learning in Digital Networks - ICT literacy: A novel assessment of students' 21st century skills , 2017, Comput. Educ..

[55]  D. Dörner,et al.  Complex Problem Solving: What It Is and What It Is Not , 2017, Front. Psychol..

[56]  Ronny Scherer,et al.  Revealing the processes of students' interaction with a novel collaborative problem solving task: An in-depth analysis of think-aloud protocols , 2017, Comput. Hum. Behav..

[57]  Andreas Hepp,et al.  The Complexity of Datafication: Putting Digital Traces in Context , 2018 .

[58]  Joke Voogt,et al.  The Twenty-First Century Curriculum: Issues and Challenges , 2018 .

[59]  Samuel Greiff,et al.  Making Use of Data for Assessments: Harnessing Analytics and Data Science , 2018 .

[60]  Denise Whitelock,et al.  Progress and Challenges for Automated Scoring and Feedback Systems for Large-Scale Assessments , 2018 .

[61]  Lenore Adie,et al.  Formative assessment and feedback using information technology , 2018 .