Practical guidelines for designing and evaluating educationally oriented recommendations

There is a need for designing educationally oriented recommendations that deal with educational goals as well as learners' preferences and context in a personalised way. They have to be both based on educators' experience and perceived as adequate by learners. This paper compiles practical guidelines to produce personalised recommendations that are meant to foster active learning in online courses. These guidelines integrate three different methodologies: i) user centred design as defined by ISO 9241-210, ii) the e-learning life cycle of personalised educational systems, and iii) the layered evaluation of adaptation features. To illustrate guidelines actual utility, generality and flexibility, the paper describes their applicability to design educational recommendations in two different contexts, which in total involved 125 educators and 595 learners. These applications show benefits for learners and educators. Following this approach, we are targeting to cope with one of the main challenges in current massive open online courses, which are expected to provide personalised education to an increasing number of students without the continuous involvement of educators in supporting learners during their course interactions. Display Omitted User centred design can eliciting recommendations based on educators' experience.The e-learning life cycle can personalise the learning experience on learners' needs.Layered evaluation can evaluate the adaptive features of the recommendations designed.Aforementioned methodologies can be jointly applied through some practical guidelines.Guidelines serve to provide personalised education in different e-learning settings.

[1]  Olga C. Santos,et al.  Extending web-based educational systems with personalised support through User Centred Designed recommendations along the e-learning life cycle , 2014, Sci. Comput. Program..

[2]  Olga C. Santos,et al.  Recommendations support in standard-based learning management systems , 2009, AIED.

[3]  S. Weibelzahl,et al.  Framework for the Evaluation of Adaptive CBR-Systems , 2001 .

[4]  Robin D. Burke,et al.  Hybrid Recommender Systems: Survey and Experiments , 2002, User Modeling and User-Adapted Interaction.

[5]  Bart P. Knijnenburg,et al.  Explaining the user experience of recommender systems , 2012, User Modeling and User-Adapted Interaction.

[6]  Olga C. Santos,et al.  TORMES Methodology to Elicit Educational Oriented Recommendations , 2011, AIED.

[7]  Hildegard Rumetshofer,et al.  Individualized e-learning systems enabled by a semantically determined adaptation of learning fragments , 2003, 14th International Workshop on Database and Expert Systems Applications, 2003. Proceedings..

[8]  Róbert Móro,et al.  ALEF: From Application to Platform for Adaptive Collaborative Learning , 2014, Recommender Systems for Technology Enhanced Learning.

[9]  Olga C. Santos,et al.  Basic Skills Training to Disabled and Adult Learners Through an Accessible e-Learning Platform , 2007, HCI.

[10]  Evandro Costa,et al.  A Systematic Approach for Designing Educational Recommender Systems , 2012 .

[11]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[12]  Constantine Stephanidis,et al.  Universal Access in Human-Computer Interaction , 2011 .

[13]  Judith Masthoff,et al.  Layered evaluation of interactive adaptive systems: framework and formative methods , 2010, User Modeling and User-Adapted Interaction.

[14]  Ian H. Witten,et al.  The WEKA data mining software: an update , 2009, SKDD.

[15]  K. Masters.  A Brief Guide To Understanding MOOCs , 2011 .

[16]  Olga C. Santos,et al.  Extending Computer Assisted Assessment Systems with Natural Language Processing, User Modeling, and Recommendations Based on Human Computer Interaction and Data Mining , 2011, IJCAI.

[17]  John Riedl,et al.  How many bits per rating? , 2012, RecSys.

[18]  R. Felder,et al.  Learning and Teaching Styles in Engineering Education. , 1988 .

[19]  Vincent P. Wade,et al.  The evaluation of adaptive and personalised information retrieval systems: a review , 2011, Int. J. Knowl. Web Intell..

[20]  Olga C. Santos,et al.  A Standards-based Modelling Approach for Dynamic Generation of Adaptive Learning Scenarios , 2008, J. Univers. Comput. Sci..

[21]  Michael J. Prince,et al.  The Effect of Different Active Learning Environments on Student Outcomes Related to Lifelong Learning , 2012 .

[22]  Peter Brusilovsky,et al.  Social Navigation Support in a Course Recommendation System , 2006, AH.

[23]  Minjuan Wang,et al.  Affective e-Learning: Using "Emotional" Data to Improve Learning in Pervasive Learning Environment , 2009, J. Educ. Technol. Soc..

[24]  Vincent P. Wade,et al.  What Do Academic Users Really Want from an Adaptive Learning System? , 2009, UMAP.

[25]  Olga C. Santos,et al.  Modeling recommendations for the educational domain , 2010, RecSysTEL@RecSys.

[26]  Vincent P. Wade,et al.  Service-Oriented E-Learning Platforms: From Monolithic Systems to Flexible Services , 2007, IEEE Internet Computing.

[27]  Cristina Gena,et al.  A User-Centered Approach for Adaptive Systems Evaluation , 2006 .

[28]  Peter Brusilovsky,et al.  Layered evaluation of adaptive learning systems , 2004 .

[29]  Gordon I. McCalla,et al.  A Multidimensional Paper Recommender: Experiments and Evaluations , 2009, IEEE Internet Computing.

[30]  Olga C. Santos,et al.  Requirements for Semantic Educational Recommender Systems in Formal E-Learning Scenarios , 2011, Algorithms.

[31]  Hendrik Drachsler,et al.  Identifying the Goal, User model and Conditions of Recommender Systems for Formal and Informal Learning , 2009, J. Digit. Inf..

[32]  Sergio Salmeron-Majadas,et al.  A Methodological Approach to Eliciting Affective Educational Recommendations , 2014, 2014 IEEE 14th International Conference on Advanced Learning Technologies.

[33]  Sebastián Ventura,et al.  Personalized Links Recommendation Based on Data Mining in Adaptive Educational Hypermedia Systems , 2007, EC-TEL.

[34]  Demetrios G. Sampson,et al.  LAYERED EVALUATION IN RECOMMENDER SYSTEMS: A RETROSPECTIVE , 2014 .

[35]  Jürgen Buder,et al.  Learning with personalized recommender systems: A psychological view , 2012, Comput. Hum. Behav..

[36]  Nigel Bevan,et al.  UsabilityNet Methods for User Centred Design , 2003 .

[37]  John Riedl,et al.  Recommender systems: from algorithms to user experience , 2012, User Modeling and User-Adapted Interaction.

[38]  Cristina Gena,et al.  Usability Engineering for the Adaptive Web , 2007, The Adaptive Web.

[39]  Cole Shaw,et al.  An Asynchronous, Personalized Learning Platform―Guided Learning Pathways (GLP) , 2014 .

[40]  Rong Zheng,et al.  REQUEST: A Query Language for Customizing Recommendations , 2011, Inf. Syst. Res..

[41]  Jan Gulliksen,et al.  Key principles for user-centred systems design , 2003, Behav. Inf. Technol..

[42]  Nishikant Sonwalkar,et al.  The First Adaptive MOOC: A Case Study on Pedagogy Framework and Scalable Cloud Architecture—Part I , 2013 .

[43]  Sergio Salmeron-Majadas,et al.  Providing Personalized Guidance in Arithmetic Problem Solving , 2014, UMAP Workshops.

[44]  S. Graf,et al.  Adaptive and Intelligent Web-Based Educational Systems , 2009 .

[45]  Jesus G. Boticario,et al.  Educational Recommender Systems and Technologies: Practices and Challenges , 2011 .

[46]  Mercedes Gómez-Albarrán,et al.  A hybrid user-centred recommendation strategy applied to repositories of learning objects , 2012, Int. J. Web Based Communities.

[47]  Mary Beth Rosson,et al.  Usability Engineering: Scenario-based Development of Human-Computer Interaction , 2001 .

[48]  Katrien Verbert,et al.  Panorama of Recommender Systems to Support Learning , 2015, Recommender Systems Handbook.

[49]  Jesus G. Boticario,et al.  Catering for Personal Autonomy with E-Mentoring Supported by Recommendations , 2011 .

[50]  Pei-Yu Wang.  The Analysis and Design of Educational Recommender Systems , 2007 .

[51]  Jesus G. Boticario,et al.  MAMIPEC - Affective Modeling in Inclusive Personalized Educational Scenarios , 2012 .

[52]  Arne Jönsson,et al.  Wizard of Oz studies: why and how , 1993, IUI '93.

[53]  Osmar R. Zaïane,et al.  Building a Recommender Agent for e-Learning Systems , 2002, ICCE.

[54]  Alexandros Paramythis,et al.  A modular approach to the evaluation of Adaptive User Interfaces , 2001 .

[55]  Olga C. Santos,et al.  User‐centred design and educational data mining support during the recommendations elicitation process in social online learning environments , 2015, Expert Syst. J. Knowl. Eng..

[56]  Philip S. Yu,et al.  Dynamic Load Balancing on Web-Server Systems , 1999, IEEE Internet Comput..

[57]  J. Boticario,et al.  Why using dotLRN ? UNED use cases , 2007 .

[58]  Olga C. Santos,et al.  Users' Experience with a Recommender System in an Open Source Standard-Based Learning Management System , 2008, USAB.

[59]  Carlos Delgado Kloos,et al.  Enabling interoperability for LMS educational services , 2009, Comput. Stand. Interfaces.

[60]  Demetrios G. Sampson,et al.  Layered evaluation in recommender systems: A retrospective assessment , 2014 .

[61]  Roger C. Schank,et al.  Engines for Education , 1995 .

[62]  Heinz Ulrich Hoppe,et al.  Analysis Methods for Collaborative Models and Activities , 2003, CSCL.

[63]  Emmanuelle Raffenne,et al.  Accessible Lifelong Learning at Higher Education: Outcomes and Lessons Learned at two Different Pilot Sites in the EU4ALL Project , 2012, J. Univers. Comput. Sci..

[64]  Ismael Pascual-Nieto,et al.  Computer-assisted assessment of free-text answers , 2009, The Knowledge Engineering Review.

[65]  Francesco Ricci,et al.  Prototype Testing for a Destination Recommender System: Steps, Procedures and Implications , 2004, ENTER.

[66]  Li Chen,et al.  Evaluating recommender systems from the user’s perspective: survey of the state of the art , 2012, User Modeling and User-Adapted Interaction.

[67]  Ruimin Shen,et al.  Learning Content Recommendation Service Based-On Simple Sequencing Specification , 2004, ICWL.

[68]  Carlos Delgado Kloos,et al.  A Generic Architecture for Emotion-based Recommender Systems in Cloud Learning Environments , 2013, J. Univers. Comput. Sci..

[69]  Philip Barker,et al.  Using e-learning dialogues in higher education , 2004 .

[70]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[71]  Bill Bates,et al.  Supporting a Range of Learning Styles Using a Taxonomy-Based Design Framework Approach. , 2001 .

[72]  Alejandro Rodríguez-Ascaso,et al.  Usability and Accessibility Evaluations Along the eLearning Cycle , 2007, WISE Workshops.

[73]  Karel Vredenburg,et al.  The state of user-centered design practice , 2005, IEEE Engineering Management Review.

[74]  Fernando Gamboa Rodríguez,et al.  A student centered methodology for the development of a physics video based laboratory , 2001, Interact. Comput..

[75]  Chih-Kai Chang,et al.  Development of a reading material recommendation system based on a knowledge engineering approach , 2010, Comput. Educ..

[76]  Olga C. Santos,et al.  Open Issues in Educational Affective Recommendations for Distance Learning Scenarios , 2013, UMAP Workshops.

[77]  Olga C. Santos,et al.  An Approach for an Affective Educational Recommendation Model , 2014, Recommender Systems for Technology Enhanced Learning.

[78]  Olga C. Santos,et al.  Toward interactive context-aware affective educational recommendations in computer-assisted language learning , 2016, New Rev. Hypermedia Multim..

[79]  Jesus G. Boticario,et al.  The Full Life Cycle of Adaptation in aLFanet eLearning Environment , 2004 .

[80]  Jianming Dong,et al.  A user input and analysis tool for information architecture , 2001, CHI Extended Abstracts.

[81]  Jesus G. Boticario,et al.  An open IMS-based user modelling approach for developing adaptive learning management systems , 2007 .

[82]  Cristina Conati,et al.  Knowledge Elicitation Methods for Affect Modelling in Education , 2013, Int. J. Artif. Intell. Educ..

[83]  Thea van der Geest,et al.  User-centered evaluation of adaptive and adaptable systems: a literature review , 2008, The Knowledge Engineering Review.

[84]  Demetrios G. Sampson,et al.  Integrating Adaptive Educational Content into Different Courses and Curricula , 2001, J. Educ. Technol. Soc..

[85]  Rob Nadolski,et al.  Effects of the ISIS Recommender System for Navigation Support in self-organised Learning Networks , 2009, J. Educ. Technol. Soc..