Systematic Mapping Study 2012-2017: Quality and Effectiveness Measurement in MOOC

Massive Open Online Courses (MOOCs) concentrate an important number of students and are set up as an alternative for acquiring knowledge and continuing education. The objective of this article is to analyze how the quality and effectiveness of the MOOCs were measured in empirical studies between 2012 and 2017. For which a systematic mapping study of articles was performed by using databases of Scopus and Web of Science. These articles were analyzed according to the: (1) context of the publication, (2) type of developing institutions and distribution platforms of the MOOC, and (3) characteristics of the empirical studies. It was identified that 54% of the developing institutions were universities and that 31% of the platforms that distributed the MOOC are of Coursera. These articles were also analyzed by type of study, the more frequently used indicators to measure the effectiveness, and the more frequently used focus points (approaches) to measure quality. The results of this study are useful because it allows having a general view of the most frequently utilized methodologies to measure quality and effectiveness in MOOCs.

[1]  Timothy K. Shih,et al.  Auto Grouping and Peer Grading System in Massive Open Online Course (MOOC) , 2015, Int. J. Distance Educ. Technol..

[2]  Ignacio Montero,et al.  A guide for naming research studies inPsychology , 2007 .

[3]  Jane Sinclair,et al.  A study of user participation across different delivery modes of a massive open online course , 2016, Int. J. Learn. Technol..

[4]  Olaf Zawacki-Richter,et al.  Trends and Patterns in Massive Open Online Courses: Review and Content Analysis of Research on MOOCs (2008-2015) , 2017 .

[5]  Patricia Gómez Hernández,et al.  Calidad de un MOOC sobre inclusión educativa: aplicación de varios instrumentos e indicadores , 2016 .

[6]  M. Winkleby,et al.  Massive open online nutrition and cooking course for improved eating behaviors and meal composition , 2015, International Journal of Behavioral Nutrition and Physical Activity.

[7]  Rosabel Roig Vila,et al.  Validation of the Questionnaire of Quality Assessment of Online Courses adapted to MOOC , 2015 .

[8]  Theodore Frick,et al.  MOOCs for Research: The Case of the Indiana University Plagiarism Tutorials and Tests , 2016, Technol. Knowl. Learn..

[9]  Teresa Torres-Coronas,et al.  MOOC and Blended Learning Models: Analysis from a Stakeholders' Perspective , 2017, Int. J. Inf. Commun. Technol. Educ..

[10]  Patrina Law,et al.  Recognising Informal Elearning with Digital Badging: Evidence for a Sustainable Business Model , 2015 .

[11]  Pearl Brereton,et al.  Performing systematic literature reviews in software engineering , 2006, ICSE.

[12]  Panagiotis G. Ipeirotis,et al.  A system for scalable and reliable technical-skill testing in online labor markets , 2015, Comput. Networks.

[13]  Miguel Baldomero Ramírez-Fernández.  La valoración de MOOC: una perspectiva de Calidad MOOC appraisal: A quality perspective , 2015 .

[14]  S A Chapman,et al.  A strategy for monitoring and evaluating massive open online courses. , 2016, Evaluation and program planning.

[15]  Jane Sinclair,et al.  Student engagement in massive open online courses , 2016, Int. J. Learn. Technol..

[16]  K. Jablokow,et al.  How much does student engagement with videos and forums in a MOOC affect their achievement , 2017 .

[17]  Ulrik Schroeder,et al.  MOOCs - A Review of the State-of-the-Art , 2014, CSEDU.

[18]  Jonathan Hughes,et al.  Assessment worlds colliding? Negotiating between discourses of assessment on an online open course , 2016 .

[19]  Patsy Moskal,et al.  Enrollment, Engagement, and Satisfaction in the BlendKit Faculty Development Open, Online Course. , 2015 .

[20]  Juan Arias Masa.  Evaluación de la calidad de cursos virtuales : Indicadores de calidad y construcción de un cuestionario de medida. Aplicación al ámbito de asignaturas de ingeniería telemática , 2008 .

[21]  Sarah E. M. Meek,et al.  Is peer review an appropriate form of assessment in a MOOC? Student participation and performance in formative peer review , 2017 .

[22]  J. Greene,et al.  Predictors of Retention and Achievement in a Massive Open Online Course , 2015 .

[23]  George Siemens,et al.  The Open Course: Through the Open Door--Open Courses as Research, Learning, and Engagement , 2010 .

[24]  Shirley Williams,et al.  MOOCs: A systematic study of the published literature 2008-2012 , 2013 .

[25]  Inmaculada Maiz Olazabalaga,et al.  Design, Motivation and Performance in a Cooperative MOOC Course , 2015 .

[26]  Venkataraman Balaji,et al.  Making Sense of MOOCs: A Guide for Policy-Makers in Developing Countries , 2016 .

[27]  Indika Perera,et al.  Quality of MOOCs: A review of literature on effectiveness and quality aspects , 2015, 2015 8th International Conference on Ubi-Media Computing (UMEDIA).

[28]  Yuan Yao,et al.  Reflections on a massive open online life cycle assessment course , 2014, The International Journal of Life Cycle Assessment.

[29]  F. Langford,et al.  Massive Open Online Courses as a Tool for Global Animal Welfare Education. , 2016, Journal of veterinary medical education.

[30]  Tiejun Zhu.  Empirical Research on the Effectiveness of MOOCs in Developing Design Students on Sino-foreign Cooperation University Programs , 2016, iJET.

[31]  Lorena Alemán de la Garza,et al.  Indicators of pedagogical quality for the design of a Massive Open Online Course for teacher training , 2015, International Journal of Educational Technology in Higher Education.

[32]  Meltem Huri Baturay.  An Overview of the World of MOOCs , 2015 .

[33]  Muhammad Farhan,et al.  An Interactive Assessment Framework for Visual Engagement: Statistical Analysis of a TEDx Video , 2017 .

[34]  Jose Luis Salmeren Silvera,et al.  Comparative between quality assessment tools for MOOCs: ADECUR vs Standard UNE 66181: 2012 , 2015, International Journal of Educational Technology in Higher Education.

[35]  Kannan M. Moudgalya,et al.  Comparing the Effectiveness of Self-Learning Java Workshops with Traditional Classrooms , 2016, J. Educ. Technol. Soc..

[36]  Ismene Ithaí Bras Ruiz.  Los MOOC en números: un análisis para comenzar la reflexión , 2016 .

[37]  Jing Chen,et al.  Research on Vocational Tendency and Learning Quality , 2017 .

[38]  Zhi Liu,et al.  Sentiment recognition of online course reviews using multi-swarm optimization-based selected features , 2016, Neurocomputing.

[39]  José Luis Salmerón Silvera,et al.  EDUTOOL ® : UN INSTRUMENTO PARA LA EVALUACIÓN Y ACREDITACIÓN DE LA CALIDAD DE LOS MOOCS , 2015 .

[40]  Hedieh Najafi,et al.  A Comparison of Learner Intent and Behaviour in Live and Archived MOOCs. , 2014 .

[41]  Elsa Loftis,et al.  Developing Online Information Literacy Instruction for the Undergraduate Art Student: A Collaborative Approach in the Context of the Framework for Information Literacy , 2016, Art Documentation: Journal of the Art Libraries Society of North America.

[42]  Rafi Nachmias,et al.  Types of Participant Behavior in a Massive Open Online Course , 2017 .

[43]  George Veletsianos,et al.  A Systematic Analysis and Synthesis of the Empirical MOOC Literature Published in 2013–2015 , 2016 .

[44]  José-Andrés Martínez-Silva,et al.  Literatura y práctica: una revisión crítica acerca de los MOOC Literature and Practice: A Critical Review of MOOCs , 2015 .

[45]  Kai Petersen,et al.  Systematic Mapping Studies in Software Engineering , 2008, EASE.

[46]  Ulrik Schroeder,et al.  A Usability Evaluation of a Blended MOOC Environment: An Experimental Case Study. , 2015 .

[47]  Patrizia Ghislandi,et al.  “The fun they had” or about the quality of MOOC , 2016 .

[48]  Meina Zhu,et al.  A systematic review of research methods and topics of the empirical MOOC literature (2014-2016) , 2018, Internet High. Educ..

[49]  Marina Romeo,et al.  Quality indicators: developing “MOOCs” in the European Higher Education Area , 2016 .

[50]  Julio Cabero Almenara,et al.  Descripción de un instrumento didáctico para el análisis de modelos y estrategias de enseñanza de cursos universitarios en la red (A.D.E.C.U.R.). , 2009 .

[51]  Cristóbal Suárez Guerrero,et al.  Evaluación de la calidad pedagógica de los MOOC , 2014 .

[52]  Jukrin Moon,et al.  Design of engineering courses as a service: emotions,senses and implementation , 2017 .

[53]  Maria J Grant,et al.  A typology of reviews: an analysis of 14 review types and associated methodologies. , 2009, Health information and libraries journal.

[54]  Carlos Delgado Kloos,et al.  Evaluation of a learning analytics application for open edX platform , 2017, Comput. Sci. Inf. Syst..

[55]  Susan E. Alcock,et al.  Archaeology and the MOOC: Massive, open, online, and opportunistic , 2016 .

[56]  Marlene A. Smith.  Output from Statistical Predictive Models as Input to eLearning Dashboards , 2015, Future Internet.

[57]  Martin Ebner,et al.  Evaluation Grid for xMOOCs , 2015, iJET.

[58]  Colin Gray.  Exploring Measures of Engagement in Open Online Work Based Learning , 2015 .

[59]  Aras Bozkurt,et al.  Research Trends in Massive Open Online Course (MOOC) Theses and Dissertations: Surfing the Tsunami Wave , 2016 .

[60]  Miguel Baldomero Ramírez-Fernández.  The MECD Quality Certification Proposal of MOOC Courses , 2015 .

[61]  Allison Littlejohn,et al.  Instructional quality of Massive Open Online Courses (MOOCs) , 2015, Comput. Educ..

[62]  Eric J. Russell,et al.  The Role of Servant Leadership in Faculty Development Programs: A Review of the Literature , 2012 .

[63]  Olaf Zawacki-Richter,et al.  What Research Says About MOOCs – An Explorative Content Analysis , 2018 .

[64]  Raine Isaksson,et al.  Sustaining Sweden’s competitive position: lean lifelong learning , 2015 .

[65]  Paul Salvador Inventado,et al.  An Online Learning Collaboratory to Address Multidisciplinary Learning Challenges at Scale , 2017, IxD&A.

[66]  Tone Kvernbekk,et al.  On the Problems of Asking for a Definition of Quality in Education , 2011 .

[67]  Carlos Delgado Kloos,et al.  Detection and Evaluation of Emotions in Massive Open Online Courses , 2015, J. Univers. Comput. Sci..

[68]  Begoña E. Sampedro-Requena,et al.  Research contributions on the educational use of MOOCs , 2017 .

[69]  J. Vickers,et al.  Relationship between participants’ level of education and engagement in their completion of the Understanding Dementia Massive Open Online Course , 2015, BMC medical education.

[70]  Matthew Leach,et al.  Supporting, categorising and visualising diverse learner behaviour on MOOCs with modular design and micro-learning , 2017, J. Comput. High. Educ..

[71]  Josep M. Duart,et al.  The pedagogical quality of MOOCs based on a systematic review of JCR and Scopus publications (2013-2015) , 2017 .

[72]  Abdellatif Medouri,et al.  Massive Open Online Courses : A New Dawn for Higher Education? , 2013 .

[73]  Carlos Delgado Kloos,et al.  Flipping the classroom to improve learning with MOOCs technology , 2017, Comput. Appl. Eng. Educ..

[74]  Javier P. Gisbert,et al.  ¿Cómo realizar, evaluar y utilizar revisiones sistemáticas y metaanálisis? , 2004 .

[75]  Ming-Yuan Hsieh,et al.  Online Learning Era: Exploring the Most Decisive Determinants of MOOCs in Taiwanese Higher Education , 2016 .

[76]  Carlos Delgado Kloos,et al.  An Algorithm for Peer Review Matching in Massive Courses for Minimising Students' Frustration , 2013, J. Univers. Comput. Sci..

[77]  Rose Alinda Alias,et al.  Nascent research trends in MOOCs in higher educational institutions: A systematic literature review , 2014, 2014 International Conference on Web and Open Access to Learning (ICWOAL).