An Evaluation Framework for Data Competitions in TEL

This paper presents a study describing the development of an Evaluation Framework (EF) for data competitions in TEL. The study applies the Group Concept Method (GCM) to empirically depict criteria and their indicators for evaluating software applications in TEL. A statistical analysis including multidimensional scaling and hierarchical clustering on the GCM data identified the following six evaluation criteria: 1.Educational Innovation, 2.Usability, 3.Data, 4.Performance, 5.Privacy, and 6.Audience. Each of them was operationalized through a set of indicators. The resulting Evaluation Framework (EF) incorporating these criteria was applied to the first data competition of the LinkedUp project. The EF was consequently improved using the results from reviewers' interviews, which were analysed qualitatively and quantitatively. The outcome of these efforts is a comprehensive EF that can be used for TEL data competitions and for the evaluation of TEL tools in general.

[1]  A. Strauss,et al.  Grounded theory methodology: An overview. , 1994 .

[2]  G. Murphy,et al.  Concept mapping for planning and evaluation , 2007, Journal of Management & Organization.

[3]  Erik Duval,et al.  Dataset-driven research for improving recommender systems for learning , 2011, LAK.

[4]  Fridolin Wild,et al.  Grand Challenge Problems from the Alpine Rendez-Vous: An Introduction , 2014 .

[5]  Mary Kane,et al.  Concept Mapping for Planning and Evaluation , 2006 .

[6]  Peter Brusilovsky,et al.  A Study of User Model Based Link Annotation in Educational Hypermedia , 1998, J. Univers. Comput. Sci..

[7]  Alexandros Paramythis,et al.  A modular approach to the evaluation of Adaptive User Interfaces , 2001 .

[8]  Taibi Davide,et al.  Using linked data in Learning Analytics , 2014 .

[9]  Katrien Verbert,et al.  Preface [Special issue on dataTEL – Data Supported Research in Technology-Enhanced Learning] , 2012 .

[10]  Erik Duval,et al.  Recommender Systems for Learning , 2012, Springer Briefs in Electrical and Computer Engineering.

[11]  Erik Duval,et al.  Dataset-Driven Research to Support Learning and Knowledge Analytics , 2012, J. Educ. Technol. Soc..

[12]  SunPei-Chen,et al.  What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction , 2008 .

[13]  Panagiotis Zaharias,et al.  Minimizing Attrition in Online Degree Courses. , 2009 .

[14]  Andrina Granic,et al.  Usability Testing and Expert Inspections Complemented by Educational Evaluation: A Case Study of an e-Learning Platform , 2011, J. Educ. Technol. Soc..

[15]  Peter Brusilovsky,et al.  Layered evaluation of adaptive learning systems , 2004 .

[16]  Dowming Yeh,et al.  What drives a successful e-Learning? An empirical investigation of the critical factors influencing learner satisfaction , 2008, Comput. Educ..

[17]  Mary Kane,et al.  Quality and rigor of the concept mapping methodology: a pooled study analysis. , 2012, Evaluation and program planning.

[18]  Panagiotis Zaharias,et al.  Usability in the Context of e-Learning: A Framework Augmenting 'Traditional' Usability Constructs with Instructional Design and Motivation to Learn , 2009, Int. J. Technol. Hum. Interact..

[19]  Stephan Weibelzahl,et al.  Evaluation of Adaptive Systems , 2001, User Modeling.

[20]  Thomas C. Reeves,et al.  Usability and Instructional Design Heuristics for E-Learning Evaluation , 2002 .