Multi-criteria decision making approach in E-learning: A systematic review and classification

Display Omitted A review regarding application of MCDM techniques in E-learning is conducted.Five major databases have been considered for this review.ScienceDirect, Emerald, Taylor and Francis, IEEE, and Springer have been selected.A systematic and meta analysis methodology called "PRISMA" has been proposed.Papers between 2001 and 2015 have been considered for this review. Academic research on E-learning has increased extensively over the past few years. Although, many multi-criteria decision making methods have been proposed to evaluate and examine the effectiveness of E-learning, there is a lack of study concerning systematic literature review and classification of research in this area. Regarding this, five major databases including ScienceDirect, Emerald, Taylor and Francis, IEEE, and Springer have been selected and a systematic methodology proposed. Consequently, a review of 42 published papers appearing in 33 academic journals and international conferences between 2001 and 2015 have been obtained to achieve a comprehensive review of MCDM application in E-learning. Accordingly, the selected papers have been classified by the year of publication, MCDM techniques, and journals and conferences in which they appeared. In addition, the significant criteria in evaluating E-learning were found. This study supports researchers and practitioners in effectively adopting MCDM techniques regarding E-learning evaluation and provides an insight into its state-of-the-art.

[1]  Xiaohui Liu,et al.  The effects of individual differences on e-learning users' behaviour in developing countries: A structural equation model , 2014, Comput. Hum. Behav..

[2]  Radhika Santhanam,et al.  Using Self-Regulatory Learning to Enhance E-Learning-Based Information Technology Training , 2008, Inf. Syst. Res..

[3]  Kayode C. V. Ekwunife-Orakwue,et al.  The impact of transactional distance dialogic interactions on student learning outcomes in online and blended environments , 2014, Comput. Educ..

[4]  Yujiro Shimizu,et al.  Designing methods of human interface for supervisory control systems , 1998 .

[5]  P. Toth Risk of Incident Diabetes With Intensive-Dose Compared With Moderate-Dose Statin Therapy: A Meta-analysis , 2012 .

[6]  José Rui Figueira,et al.  A Multiple Criteria Decision Analysis Model Based on ELECTRE TRI-C for Erosion Risk Assessment in Agricultural Areas , 2014, Environmental Modeling & Assessment.

[7]  Matthias Ehrgott,et al.  Multiple criteria decision analysis: state of the art surveys , 2005 .

[8]  Ron Chi-Wai Kwok,et al.  Remote learning technologies: effectiveness of hypertext and GSS , 1999, Decis. Support Syst..

[9]  L.P. Dringus,et al.  An adaptable usability heuristic checklist for online courses , 2005, Proceedings Frontiers in Education 35th Annual Conference.

[10]  Ru-Jen Chao,et al.  Evaluation of the criteria and effectiveness of distance e-learning with consistent fuzzy preference relations , 2009, Expert Syst. Appl..

[11]  Ani I. Matei,et al.  E-Learning Platforms Supporting the Educational Effectiveness of Distance Learning Programme: A Comparative Study on Administrative Sciences , 2012 .

[12]  Yihui Chen,et al.  The research of evaluation system of network self-learning based on fuzzy theory , 2010, 2010 International Conference on Networking and Digital Society.

[13]  Jason K. Levy,et al.  Multiple criteria decision making and decision support systems for flood risk management , 2005 .

[14]  Selim Zaim,et al.  Selecting "The Best" ERP system for SMEs using a combination of ANP and PROMETHEE methods , 2015, Expert Syst. Appl..

[15]  Edmundas Kazimieras Zavadskas,et al.  A new additive ratio assessment (ARAS) method in multicriteria decision‐making , 2010 .

[16]  Ian Roffe,et al.  E‐learning: engagement, enhancement and execution , 2002 .

[17]  G. Dominici,et al.  How to build an e-learning product: factors for student/customer satisfaction , 2013 .

[18]  Dursun Delen,et al.  Development of a fuzzy ANP based SWOT analysis for the airline industry in Turkey , 2012, Expert Syst. Appl..

[19]  Arpan Kumar Kar,et al.  A hybrid group decision support system for supplier selection using analytic hierarchy process, fuzzy set theory and neural network , 2015, J. Comput. Sci..

[20]  N. A. Iahad,et al.  Using a Multi-Criteria Decision Making Approach for Assessing the Factors Affecting Social Network Sites Intention to Use , 2015 .

[21]  Hangjung Zo,et al.  Critical success factors for e-learning in developing countries: A comparative analysis between ICT experts and faculty , 2012, Comput. Educ..

[22]  Khaled A. Alshare,et al.  Student-Instructor Perception of Computer Technologies in Developing Countries: The Case of Jordan , 2003, J. Comput. Inf. Syst..

[23]  David Murphy,et al.  Evaluating interactivity and learning in computer conferencing using content analysis techniques , 2005 .

[24]  Hema Banati,et al.  Incorporating social opinion in content selection for an e-learning course , 2011, 2011 6th International Conference on Computer Science & Education (ICCSE).

[25]  H. Murakoshi,et al.  Comparison using AHP Web-based learning with classroom learning , 2001, Proceedings 2001 Symposium on Applications and the Internet Workshops (Cat. No.01PR0945).

[26]  S. M. Seyed Hosseini,et al.  Reprioritization of failures in a system failure mode and effects analysis by decision making trial and evaluation laboratory technique , 2006, Reliab. Eng. Syst. Saf..

[27]  Shivraj Kanungo,et al.  Appropriate media choice for e-learning effectiveness: Role of learning domain and learning style , 2014, Comput. Educ..

[28]  Angel Cobo,et al.  Evaluation of the interactivity of students in virtual learning environments using a multicriteria approach and data mining , 2014, Behav. Inf. Technol..

[29]  César Coll,et al.  Supporting online collaborative learning in small groups: Teacher feedback on learning content, academic task and social participation , 2014, Comput. Educ..

[30]  Robert P. Bostrom,et al.  A Research Framework for Collaborative eLearning in an End User Training Context , 2005, AMCIS.

[31]  Eugenijus Kurilovas,et al.  Application of Multiple Criteria Decision Analysis and Optimisation Methods in Evaluation of Quality of Learning Objects , 2011, Int. J. Online Pedagog. Course Des..

[32]  Gintautas Dzemyda,et al.  Multiple criteria decision support system: methods, user's interface and applications , 1994 .

[33]  B. Roy,et al.  The European school of MCDA: Emergence, basic features and current works , 1996 .

[34]  Mojtaba Zare,et al.  The role of Security, Design and Content factors on customer trust in mobile commerce , 2015 .

[35]  Mehrbakhsh Nilashi,et al.  Organizational decision to adopt hospital information system: An empirical investigation in the case of Malaysian public hospitals , 2015, Int. J. Medical Informatics.

[36]  Shih-Yi Kuo,et al.  The Study for selecting the consignment performance of E-learning of technology college , 2011, 2011 International Conference on Multimedia Technology.

[37]  A W Tony Bates,et al.  Technology, e-learning, and distance education , 1995 .

[38]  Hsin-Te Yeh,et al.  Team members' perceptions of online teamwork learning experiences and building teamwork trust: A qualitative study , 2013, Comput. Educ..

[39]  Gang Kou,et al.  Analytic network process in risk assessment and decision analysis , 2014, Comput. Oper. Res..

[40]  Norshidah Mohamed,et al.  E-Learning acceptance in a developing country: A case of the Indonesian Open University , 2007 .

[41]  R. Leipzig,et al.  The impact of E-learning in medical education. , 2006, Academic medicine : journal of the Association of American Medical Colleges.

[42]  Nikolaos F. Matsatsinis,et al.  A multicriteria methodology for the assessment of distance education trainees , 2005, Oper. Res..

[43]  Metin Dagdeviren,et al.  A fuzzy analytic network process (ANP) model for measurement of the sectoral competititon level (SCL) , 2010, Expert Syst. Appl..

[44]  Mehrbakhsh Nilashi,et al.  A multi-criteria approach to the evaluation of Malaysian government portal , 2012 .

[45]  Qing Zhang,et al.  The Research on Affecting Factors of E-learning Training Effect , 2008, 2008 International Conference on Computer Science and Software Engineering.

[46]  Wenli Li,et al.  Fuzzy comprehensive assessment of network environment and learning quality combined with the Analytic Hierarchy Process , 2012, 2012 2nd International Conference on Consumer Electronics, Communications and Networks (CECNet).

[47]  Tien-Hsiang Chang,et al.  Key factors for efficiently implementing customized e-learning system in the service industry , 2011 .

[48]  A. M. Abirami,et al.  A semantic based approach to organize eLearning through efficient information retrieval for interview preparation , 2012, 2012 International Conference on Recent Trends in Information Technology.

[49]  Marcin W. Mastalerz Electre method for choosing an e-learning platform , 2010, 2010 Proceedings of VIth International Conference on Perspective Technologies and Methods in MEMS Design.

[50]  Shiou-Yu Chen,et al.  Identifying and prioritizing critical intellectual capital for e‐learning companies , 2009 .

[51]  Jay F. Nunamaker,et al.  Powering E-Learning In the New Millennium: An Overview of E-Learning and Enabling Technology , 2003, Inf. Syst. Frontiers.

[52]  Nune Ravi Sankar,et al.  Modified approach for prioritization of failures in a system failure mode and effects analysis , 2001 .

[53]  Martin Oliver,et al.  An introduction to the Evaluation of Learning Technology , 2000, J. Educ. Technol. Soc..

[54]  Peter Bower,et al.  Factors affecting recruitment into depression trials: Systematic review, meta-synthesis and conceptual framework. , 2015, Journal of affective disorders.

[55]  Dennis F. Galletta,et al.  Human-Computer Interaction and Management Information Systems – Foundations , 2006 .

[56]  Jaap Spronk,et al.  Multiple Criteria Decision Methods and Applications , 1985 .

[57]  Hwa-Young Jeong,et al.  The quality model for e-learning system with multimedia contents: a pairwise comparison approach , 2013, Multimedia Tools and Applications.

[58]  Artem Feshchenko,et al.  E-learning in the Evaluation of Students and Teachers: LMS or Social Networks?☆ , 2014 .

[59]  Gwo-Hshiung Tzeng,et al.  Evaluating intertwined effects in e-learning programs: A novel hybrid MCDM model based on factor analysis and DEMATEL , 2007, Expert Syst. Appl..

[60]  Chih-Ming Chen,et al.  Assessing effects of information architecture of digital libraries on supporting E-learning: A case study on the Digital Library of Nature & Culture , 2014, Comput. Educ..

[61]  Konstantina Chrysafiadi,et al.  PeRSIVA: An empirical evaluation method of a student model of an intelligent e-learning environment for computer programming , 2013, Comput. Educ..

[62]  Abbas Keramati,et al.  Website success comparison in the context of e-recruitment: An analytic network process (ANP) approach , 2013, Appl. Soft Comput..

[63]  Cengiz Kahraman,et al.  Fuzzy Multi-Criteria Decision Making: Theory and Applications with Recent Developments , 2008 .

[64]  Rajeev Jain,et al.  A Kano model, AHP and M-TOPSIS method-based technique for disassembly line balancing under fuzzy environment , 2014, Appl. Soft Comput..

[65]  Mi Chen,et al.  Comprehensive evaluation of teaching websites based on intelligence methods , 2010, 2010 2nd IEEE International Conference on Information Management and Engineering.

[66]  Alastair D Hay,et al.  Effect of antibiotic prescribing in primary care on antimicrobial resistance in individual patients: systematic review and meta-analysis , 2010, BMJ : British Medical Journal.

[67]  Evangelos Grigoroudis,et al.  User satisfaction and e-learning systems: Towards a multi-criteria evaluation methodology , 2003, Oper. Res..

[68]  Adrien Presley,et al.  R&D project selection using the analytic network process , 2002, IEEE Trans. Engineering Management.

[69]  Hong Feng Lai,et al.  Determining the sustainability of virtual learning communities in E-learning platform , 2010, 2010 5th International Conference on Computer Science & Education.

[70]  Gülçin Büyüközkan,et al.  Evaluation of E-Learning Web Sites Using Fuzzy Axiomatic Design Based Approach , 2010, Int. J. Comput. Intell. Syst..

[71]  Sankaran Mahadevan,et al.  An evidential DEMATEL method to identify critical success factors in emergency management , 2014, Appl. Soft Comput..

[72]  M. Roizen,et al.  Technology-enhanced simulation for health professions education: a systematic review and meta-analysis , 2012 .

[73]  J. Arbaugh,et al.  A Structural Equation Model of Predictors for Effective Online Learning , 2005 .

[74]  Emilio Jiménez,et al.  Multi-attribute evaluation and selection of sites for agricultural product warehouses based on an Analytic Hierarchy Process , 2014 .

[75]  Badrul H. Khan,et al.  Web-based training: an introduction , 2001 .

[76]  M. Ramachandran,et al.  Application of multi-criteria decision making to sustainable energy planning--A review , 2004 .

[77]  Cheng Jie Evaluation and modeling of online course using fuzzy AHP , 2010, 2010 International Conference on Computer and Information Application.

[78]  Ling Chen,et al.  Critical Success Factors for Web-Based Organizational IT Training Systems , 2002, ICWL.

[79]  Murat Ince,et al.  Web-based learning object selection software using analytical hierarchy process , 2014, IET Softw..

[80]  A. Pietrosanto,et al.  Evaluation Models for E-Learning Platform: an AHP approach , 2006, Proceedings. Frontiers in Education. 36th Annual Conference.

[81]  Ralph L. Keeney,et al.  Decisions with multiple objectives: preferences and value tradeoffs , 1976 .

[82]  Herbert A. Simon,et al.  The new science of management decision , 1960 .

[83]  Eugenijus Kurilovas,et al.  New MCEQLS TFN method for evaluating quality and reusability of learning objects , 2013 .

[84]  Eugenijus Kurilovas,et al.  Multiple Criteria Evaluation of Quality and Optimisation of e-Learning System Components , 2010 .

[85]  Yi-Shun Wang,et al.  Multi-criteria evaluation of the web-based e-learning system: A methodology based on learner satisfaction and its applications , 2008, Comput. Educ..

[86]  K. K. F. Yuen,et al.  A multiple criteria decision making approach for E-Learning platform selection: the Primitive Cognitive Network Process , 2012, 2012 Computing, Communications and Applications Conference.

[87]  Jeng-Fung Chen,et al.  Evaluating teaching performance based on fuzzy AHP and comprehensive evaluation approach , 2015, Appl. Soft Comput..

[88]  J. Ioannidis,et al.  The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. , 2009, Journal of clinical epidemiology.

[89]  Zetian Fu,et al.  E-learning adoption intention and its key influence factors based on innovation adoption theory , 2010, Math. Comput. Model..

[90]  Hassan M. Selim,et al.  Critical success factors for e-learning acceptance: Confirmatory factor models , 2007, Comput. Educ..

[91]  Luca Botturi,et al.  MiLE : Systematic Usability Evaluation for E-learning Web Applications , 2004 .

[92]  Ming-Lang Tseng,et al.  Evaluating the effectiveness of e-learning system in uncertainty , 2011, Ind. Manag. Data Syst..

[93]  William Ho,et al.  Integrated analytic hierarchy process and its applications - A literature review , 2008, Eur. J. Oper. Res..

[94]  Gheorghe Grigoraş,et al.  Assessment Criteria of E-learning Environments Quality , 2014 .

[95]  Ali Jahan,et al.  Multi-criteria Decision Analysis for Supporting the Selection of Engineering Materials in Product Design , 2013 .

[96]  Francisco Rodrigues Lima Junior,et al.  A comparison between Fuzzy AHP and Fuzzy TOPSIS methods to supplier selection , 2014, Appl. Soft Comput..

[97]  Nina Begicevic,et al.  Prioritization of e-learning forms: a multicriteria methodology , 2007, Central Eur. J. Oper. Res..

[98]  Mehrbakhsh Nilashi,et al.  A Model for Detecting Customer Level Intentions to Purchase in B2C Websites Using TOPSIS and Fuzzy Logic Rule-Based System , 2013, Arabian Journal for Science and Engineering.

[99]  Qiong Liu,et al.  E-Learning Platform Evaluation Using Fuzzy AHP , 2009, 2009 International Conference on Computational Intelligence and Software Engineering.

[100]  Derek A. Holton,et al.  Students' Satisfaction and Perceived Learning with a Web-based Course , 2003, J. Educ. Technol. Soc..

[101]  Eugenijus Kurilovas,et al.  Multiple Criteria Comparative Evaluation of E-Learning Systems and Components , 2009, Informatica.

[102]  Hui-Wen Hsu,et al.  Analytic Hierarchy Process of Academic Scholars for Promoting Energy Saving and Carbon Reduction in Taiwan , 2014 .

[103]  Rosli Mohamad Zin,et al.  MCPCM: A DEMATEL-ANP-Based Multi-criteria Decision-Making Approach to Evaluate the Critical Success Factors in Construction Projects , 2015 .

[104]  N. Munkhtsetseg,et al.  Multi-criteria Comparative Evaluation of the E-Learning Systems: A Case Study , 2014, 2014 7th International Conference on Ubi-Media Computing and Workshops.

[105]  Rami Bahsoon,et al.  Measuring performance of virtual learning environment system in higher education , 2009 .

[106]  G. Tzeng,et al.  Marketing strategy based on customer behaviour for the LCD-TV , 2006 .

[107]  Amy T. Wang,et al.  Clinical review 1: Adverse effects of testosterone therapy in adult men: a systematic review and meta-analysis. , 2010, The Journal of clinical endocrinology and metabolism.

[108]  Frank S. C. Tseng,et al.  Design of an adaptive curriculum portfolio recommendation system by learning object similarity evaluation and group decision modeling , 2010, 2010 10th International Conference on Intelligent Systems Design and Applications.

[109]  Bruce L. Golden,et al.  Applications of the Analytic Hierarchy Process: A Categorized, Annotated Bibliography , 1989 .

[110]  A. K. M. Najmul Islam,et al.  Investigating e-learning system usage outcomes in the university context , 2013, Comput. Educ..

[111]  Jonathan Brinkerhoff Effects of a Long-Duration, Professional Development Academy on Technology Skills, Computer Self-Efficacy, and Technology Integration Beliefs and Practices , 2006 .

[112]  Yihui Chen,et al.  Study and construct online self-learning evaluation system model based on AHP method , 2010, 2010 2nd IEEE International Conference on Information and Financial Engineering.

[113]  Jo Salmon,et al.  Relationship between the physical environment and physical activity in older adults: a systematic review. , 2011, Health & place.

[114]  Yihui Chen,et al.  The research of evaluation system model of web self- learning based on ahp method and the system implement , 2010, 2010 International Conference on Computer Application and System Modeling (ICCASM 2010).

[115]  Francisco J. Martínez-López,et al.  Modelling students' flow experiences in an online learning environment , 2014, Comput. Educ..

[116]  Mehrbakhsh Nilashi,et al.  Evaluating the Factors Affecting Adoption of Hospital Information System Using Analytic Hierarchy Process , 2015 .

[117]  A. Gabus,et al.  Perceptions of the world problematique: communication procedure, communicating with those bearing collective responsibility , 1973 .

[118]  Eugenijus Kurilovas,et al.  Learning Objects and Virtual Learning Environments Technical Evaluation Criteria. , 2009 .

[119]  S. Michie,et al.  Using the Internet to Promote Health Behavior Change: A Systematic Review and Meta-analysis of the Impact of Theoretical Basis, Use of Behavior Change Techniques, and Mode of Delivery on Efficacy , 2010, Journal of medical Internet research.

[120]  Khitam Shraim,et al.  An e-learning approach to secondary education in Palestine: opportunities and challenges , 2010, Inf. Technol. Dev..

[121]  Cevriye Gencer,et al.  Analytic network process in supplier selection: A case study in an electronic firm , 2007 .

[122]  Eugenijus Kurilovas,et al.  New MCEQLS AHP method for evaluating quality of learning scenarios , 2013 .

[123]  Jay F. Nunamaker,et al.  Can e-learning replace classroom learning? , 2004, CACM.

[124]  Gwo-Jen Hwang,et al.  A group-decision approach for evaluating educational web sites , 2004, Comput. Educ..

[125]  Mehrbakhsh Nilashi,et al.  Prioritizing critical factors to successful adoption of total hospital information system , 2015 .

[126]  Marcia J. Simmering,et al.  E-learning: emerging uses, empirical results and future directions , 2003 .

[127]  Hsiu-Fen Lin,et al.  An application of fuzzy AHP for evaluating course website quality , 2010, Comput. Educ..

[128]  S. Gulati Technology-Enhanced Learning in Developing Nations: A Review. , 2008 .

[129]  S. Greco,et al.  ELECTRE Methods: Main Features and Recent Developments , 2010 .

[130]  Jyrki Wallenius,et al.  Multiple Criteria Decision Making: From Early History to the 21st Century , 2011 .

[131]  Numan Çelebi,et al.  Comparative analysis of multi-criteria decision making methodologies and implementation of a warehouse location selection problem , 2011, Expert Syst. Appl..

[132]  Chen-Shu Wang,et al.  Combining Fuzzy AHP and Association Rule to Evaluate the Activity Processes of E-learning System , 2012, 2012 Sixth International Conference on Genetic and Evolutionary Computing.

[133]  Liz Kolb,et al.  Adventures with Cell Phones. , 2011 .

[134]  Paul Welsh,et al.  Risk of incident diabetes with intensive-dose compared with moderate-dose statin therapy: a meta-analysis. , 2011, JAMA.

[135]  Ellen D. Wagner Emerging learning trends and the World Wide Web , 2001 .

[136]  Z. Berge,et al.  Student barriers to online learning: A factor analytic study , 2005 .

[137]  H. Buchwald,et al.  Systematic Review and Meta-analysis of Medium-Term Outcomes After Banded Roux-en-Y Gastric Bypass , 2014, Obesity Surgery.

[138]  Luis G. Vargas,et al.  The theory of ratio scale estimation: Saaty's analytic hierarchy process , 1987 .

[139]  G. M. Kundi,et al.  From E-Learning 1.0 to E-Learning 2.0: Threats & Opportunities for Higher Education Institutions in the Developing Countries , 2014 .

[140]  S. Anita Shanthi,et al.  Similarity Measure of Interval Valued Intuitionistic Fuzzy Soft Sets of Root Type in Decision Making , 2015 .

[141]  T. Urdan,et al.  Corporate elearning: exploring a new frontier , 2000 .

[142]  Rosanna Abbate,et al.  Accruing evidence on benefits of adherence to the Mediterranean diet on health: an updated systematic review and meta-analysis. , 2010, The American journal of clinical nutrition.

[143]  Kenneth L. Miller,et al.  Theoretical and Practical Considerations in the Design of Web-Based Instruction , 2000 .

[144]  Tzu-Hua Wang,et al.  Developing an assessment-centered e-Learning system for improving student learning effectiveness , 2014, Comput. Educ..

[145]  Varun Gupta,et al.  Incremental development & revolutions of E-learning software systems in education sector: a case study approach , 2013, Human-centric Computing and Information Sciences.

[146]  Benjamin A. Spencer,et al.  Beyond the Black Box: A Systematic Review of Breast, Prostate, Colorectal, and Cervical Screening Among Native and Immigrant African-Descent Caribbean Populations , 2015, Journal of Immigrant and Minority Health.

[147]  Juan Alfonso Lara,et al.  A system for knowledge discovery in e-learning environments within the European Higher Education Area - Application to student data from Open University of Madrid, UDIMA , 2014, Comput. Educ..

[148]  S. Emre Alptekin,et al.  An integrated decision framework for evaluating and selecting e-learning products , 2011, Appl. Soft Comput..

[149]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement , 2009, BMJ.

[150]  Li Bo,et al.  Modeling of Network Education Effectiveness Evaluation in Fuzzy Analytic Hierarchy Process , 2009, 2009 International Conference on Networking and Digital Society.

[151]  Åke Grönlund,et al.  A mobile e-learning environment for developing countries: the Bangladesh Virtual Interactive Classroom , 2010, Inf. Technol. Dev..

[152]  Gwo-Hshiung Tzeng,et al.  AHP, Fuzzy Measure and Fuzzy Integral Approaches for the Appraisal of Information Service Providers in Taiwan , 2003 .