The quality of experimental designs in mobile learning research: A systemic review and self-improvement tool

Abstract The intervention effects of mobile learning have become one of the most popular topics in education research. Despite scholars critiquing the quality of mobile learning research, there is an obvious lack of quantitative studies exploring the design quality of experimental studies of mobile learning as well as tools for self-checking and improving experimental design. Based on the newly developed CREED (Checklist for the Rigor of Education-Experiment Designs), which focuses on the internal, construct, and statistical-conclusion validities of research, this study investigated the experimental research designs of mobile-learning studies. By analyzing 342 experimental studies published in refereed journals from 2006 to 2016, this study found that 72% of the mobile-learning studies exhibited experimental designs with a low or medium-low level of rigor. The mobile-learning research suffered from methodological limitations such as a lack of baseline equivalence between the experimental and control groups, not examining the fulfillment of statistical assumptions for specific statistical methods, poor quality of outcome assessment tools, and insufficient statistical powers. This study concluded that while utilizing advanced technologies, mobile-learning research employs laggard methodologies compared with standards proposed in educational and psychological research. Recommendations for overcoming these limitations are provided for use in further research and practices.

[1]  Mike Sharples,et al.  The design of personal mobile technologies for lifelong learning , 2000, Comput. Educ..

[2]  Philip C. Abrami,et al.  What Forty Years of Research Says About the Impact of Technology on Learning , 2011 .

[3]  Fred Paas,et al.  Effects of mode of target task selection on learning about plants in a mobile learning environment: effortful manual selection versus effortless QR-code selection , 2016 .

[4]  Kurt Squire,et al.  Environmental Detectives: PDAs as a window into a virtual simulated world , 2002, Proceedings. IEEE International Workshop on Wireless and Mobile Technologies in Education.

[5]  G. Glass,et al.  Consequences of Failure to Meet Assumptions Underlying the Fixed Effects Analyses of Variance and Covariance , 1972 .

[6]  Y. Lincoln,et al.  Scientific Research in Education , 2004 .

[7]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[8]  Lanqin Zheng,et al.  Exploring the behavioral patterns of Co-regulation in mobile computer-supported collaborative learning , 2016, Smart Learning Environments.

[9]  Yen-Ting Lin,et al.  Mobile Seamless Technology Enhanced CSL Oral Communication , 2016, J. Educ. Technol. Soc..

[10]  Yao-Ting Sung,et al.  Using mobile devices to enhance the interactive learning for spatial geometry , 2016, Interact. Learn. Environ..

[11]  Luis Miller,et al.  Why a Trade-Off? The Relationship between the External and Internal Validity of Experiments , 2010, THEORIA.

[12]  Kuo-En Chang,et al.  How effective are mobile devices for language learning? A meta-analysis , 2015 .

[13]  Marco Kalz,et al.  A review of the types of mobile activities in mobile inquiry-based learning , 2018, Comput. Educ..

[14]  Daniël Lakens,et al.  Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs , 2013, Front. Psychol..

[15]  Douglas G Altman,et al.  Systematic reviews in health care: Assessing the quality of controlled clinical trials. , 2001, BMJ.

[16]  Mark Warschauer,et al.  Digital Writing and Diversity: The Effects of School Laptop Programs on Literacy Processes and Outcomes , 2013 .

[17]  Gwo-Jen Hwang,et al.  Trends and research issues of mobile learning studies in hospitality, leisure, sport and tourism education: a review of academic publications from 2002 to 2017 , 2018, Interact. Learn. Environ..

[18]  Chun-Yen Chang,et al.  Beyond the Flipped Classroom: A Highly Interactive Cloud-Classroom (HIC) Embedded into Basic Materials Science Courses , 2016 .

[19]  B. Tabachnick,et al.  Using Multivariate Statistics , 1983 .

[20]  Clayton R. Cook,et al.  Social Skills Training for Children and Youth with Emotional and Behavioral Disorders: Validity Considerations and Future Directions , 2004 .

[21]  Ming-Yueh Hwang,et al.  An Exploration of Students’ Science Learning Interest Related to Their Cognitive Anxiety, Cognitive Load, Self-Confidence and Learning Progress Using Inquiry-Based Learning With an iPad , 2017 .

[22]  Kinshuk,et al.  An investigation of attitudes of students and teachers about participating in a context-aware ubiquitous learning activity , 2011 .

[23]  Faria Sana,et al.  Laptop multitasking hinders classroom learning for both users and nearby peers , 2013, Comput. Educ..

[24]  M. Warschauer,et al.  Learning in One-to-One Laptop Environments , 2016 .

[25]  David Parsons,et al.  Abductive science inquiry using mobile devices in the classroom , 2013, Comput. Educ..

[26]  A. Aron,et al.  Statistics for Psychology , 1994 .

[27]  Harris Cooper,et al.  A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: the Study Design and Implementation Assessment Device (Study DIAD). , 2008, Psychological methods.

[28]  K. Hirsh-Pasek,et al.  Living in Pasteur’s Quadrant: How Conversational Duets Spark Language at Home and in the Community , 2018 .

[29]  Berlin Wu,et al.  Mobile Game-Based Learning with a Mobile App: Motivational Effects and Learning Performance , 2017, J. Adv. Comput. Intell. Intell. Informatics.

[30]  Yu-Hui Ching,et al.  Mobile computer-supported collaborative learning: A review of experimental research , 2013, Br. J. Educ. Technol..

[31]  Martin Boeker,et al.  Game-Based E-Learning Is More Effective than a Conventional Instructional Method: A Randomized Controlled Trial with Third-Year Medical Students , 2013, PloS one.

[32]  Robert E. Slavin,et al.  How features of educational technology applications affect student reading outcomes: A meta-analysis , 2012 .

[33]  Eva García,et al.  An empirical study on m-learning adaptation: Learning performance and learning contexts , 2015, Comput. Educ..

[34]  Nian-Shing Chen,et al.  EFFECTS OF SHORT-TERM MEMORY AND CONTENT REPRESENTATION TYPE ON MOBILE LANGUAGE LEARNING , 2008 .

[35]  Chris Evans,et al.  The effectiveness of m-learning in the form of podcast revision lectures in higher education , 2008, Comput. Educ..

[36]  R. Slavin.  Evidence-Based Education Policies: Transforming Educational Practice and Research , 2002 .

[37]  D. C. Howell.  Statistical Methods for Psychology , 1987 .

[38]  Chin-Chung Tsai,et al.  Chinese undergraduate students' perceptions of mobile learning: Conceptions, learning profiles, and approaches , 2019, J. Comput. Assist. Learn..

[39]  Helen Crompton,et al.  The use of mobile learning in PK-12 education: A systematic review , 2017, Comput. Educ..

[40]  Ozlem Baydas,et al.  Educational technology research trends from 2002 to 2014 , 2015, Scientometrics.

[41]  Yanjie Song,et al.  Methodological Issues in Mobile Computer-Supported Collaborative Learning (mCSCL): What Methods, What to Measure and When to Measure? , 2014, J. Educ. Technol. Soc..

[42]  Chih-Cheng Lin,et al.  Learning English reading in a mobile-assisted extensive reading program , 2014, Comput. Educ..

[43]  Yen-Ting Lin,et al.  The effects of mobile learning on students' oral performance in Mandarin Chinese and their attitudes , 2016, Int. J. Mob. Learn. Organisation.

[44]  Ciara Keenan,et al.  The trials of evidence-based practice in education: a systematic review of randomised controlled trials in education research 1980–2016 , 2018, Educational Research.

[45]  A. Stone.  Editorial: Modification to “Instructions to authors”. , 2003 .

[46]  Anders Jonsson,et al.  The use of scoring rubrics: Reliability, validity, and educational consequences , 2007 .

[47]  Miguel Nussbaum,et al.  Computer supported collaborative learning using wirelessly interconnected handheld computers , 2004, Comput. Educ..

[48]  Gwo-Jen Hwang,et al.  Research trends in mobile and ubiquitous learning: a review of publications in selected journals from 2001 to 2010 , 2011, Br. J. Educ. Technol..

[49]  Chun-Yi Wu,et al.  Application of context-aware and personalized recommendation to implement an adaptive ubiquitous learning system , 2011, Expert Syst. Appl..

[50]  S. Raudenbush.  Statistical analysis and optimal design for cluster randomized trials , 1997 .

[51]  Tan-Hsu Tan,et al.  Outdoor Natural Science Learning with an RFID-Supported Immersive Ubiquitous Learning Environment , 2009, J. Educ. Technol. Soc..

[52]  George M. Chinnery.  EMERGING TECHNOLOGIES Going to the MALL: Mobile Assisted Language Learning , 2006 .

[53]  Juan Ramón Pérez Pérez,et al.  Music learning in preschool with mobile devices , 2017, Behav. Inf. Technol..

[54]  Nuray Gedik,et al.  Research trends in mobile assisted language learning from 2000 to 2012 , 2014, ReCALL.

[55]  Sarah E. Ainsworth,et al.  An Active-Learning Approach to Fostering Understanding of Research Methods in Large Classes , 2017 .

[56]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[57]  Luke Ramsay,et al.  Can a tablet device alter undergraduate science students' study behavior and use of technology? , 2012, Advances in physiology education.

[58]  M. Lipsey,et al.  The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. , 1993, American Psychologist.

[59]  Random Thoughts on the (In)credibility of Educational-Psychological Intervention Research , 2004 .

[60]  Gwo-Jen Hwang,et al.  Effects of a Structured Resource-based Web Issue-Quest Approach on Students' Learning Performances in Computer Programming Courses , 2017, J. Educ. Technol. Soc..

[61]  E. Erdfelder,et al.  Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses , 2009, Behavior research methods.

[62]  Erkki Sutinen,et al.  The effectiveness of automatic text summarization in mobile learning contexts , 2013, Comput. Educ..

[63]  Chiu-Lin Lai,et al.  Trends and research issues of mobile learning studies in nursing education: A review of academic publications from 1971 to 2016 , 2018, Comput. Educ..

[64]  Kinshuk,et al.  Integration of Mobile AR Technology in Performance Assessment , 2016, J. Educ. Technol. Soc..

[65]  Lung-Hsiang Wong,et al.  What seams do we remove in mobile-assisted seamless learning? A critical review of the literature , 2011, Comput. Educ..

[66]  Chin-Chung Tsai,et al.  Preservice teachers' conceptions of teaching using mobile devices and the quality of technology integration in lesson plans , 2019, Br. J. Educ. Technol..

[67]  Tzu-Chien Liu,et al.  The modality effect in a mobile learning environment: Learning from spoken text and real objects , 2019, Br. J. Educ. Technol..

[68]  Yao-Ting Sung,et al.  The Effects of Mobile-Computer-Supported Collaborative Learning: Meta-Analysis and Critical Synthesis , 2017, Review of educational research.

[69]  Tony Gentry,et al.  PDAs as cognitive aids for people with multiple sclerosis. , 2008, The American journal of occupational therapy : official publication of the American Occupational Therapy Association.

[70]  K. E. Barron,et al.  Debating Curricular Strategies for Teaching Statistics and Research Methods , 2014 .

[71]  F. Paas,et al.  Effects of spell checkers on English as a second language students’ incidental spelling learning: a cognitive load perspective , 2017 .

[72]  Jin Kyoung Hwang,et al.  Building word knowledge, learning strategies, and metacognition with the Word-Knowledge e-Book , 2019, Comput. Educ..

[73]  Cynthia S. Puranik,et al.  Using tablet computers to teach preschool children to write letters: Exploring the impact of extrinsic and intrinsic feedback , 2016, Comput. Educ..

[74]  Magdalena Mateescu,et al.  Mobile and ubiquitous learning in higher education settings. A systematic review of empirical studies , 2016, Comput. Hum. Behav..

[75]  Abraham E. Flanigan,et al.  What College Instructors Can Do About Student Cyber-slacking , 2017, Educational Psychology Review.

[76]  R. Slavin.  A Reader's Guide to Scientifically Based Research. , 2003 .

[77]  Yao-Ting Sung,et al.  Designing an electronic guidebook for learning engagement in a museum of history , 2010, Comput. Hum. Behav..

[78]  Jack Burston,et al.  Twenty years of MALL project implementation: A meta-analysis of learning outcomes , 2014, ReCALL.

[79]  Robert F. Boruch,et al.  Randomized Experiments for Planning and Evaluation: A Practical Guide , 1998 .

[80]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[81]  Teaching evidence-based practice: implications for psychology. , 2007, Journal of clinical psychology.

[82]  J M Bos,et al.  Using Cluster Random Assignment to Measure Program Impacts , 1999, Evaluation review.

[83]  G. Norman,et al.  Randomized controlled trials. , 2004, AJR. American journal of roentgenology.

[84]  Rebecca Herman,et al.  Critical Issues and Common Pitfalls in Designing and Conducting Impact Studies in Education , 2010 .

[85]  Murat Tezer,et al.  The impact of using mobile-supported learning management systems in teaching web design on the academic success of students and their opinions on the course , 2018, Interact. Learn. Environ..

[86]  Norman E. Wallen,et al.  How to Design and Evaluate Research in Education , 1990 .

[87]  D. Moher,et al.  The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials , 2001, The Lancet.

[88]  Angela Boland,et al.  Doing a Systematic Review. A Student's Guide. Third Edition , 2023, Revue d'Épidémiologie et de Santé Publique.

[89]  Misganaw T. Gebru,et al.  Effect of Clickers "versus" Online Homework on Students' Long-Term Retention of General Chemistry Course Material , 2012 .

[90]  T H Ollendick,et al.  Empirically supported psychological interventions: controversies and evidence. , 2001, Annual review of psychology.

[91]  Reza Dashtestani,et al.  Moving bravely towards mobile learning: Iranian students' use of mobile devices for learning English as a foreign language , 2016 .

[92]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[93]  Alan C. K. Cheung,et al.  The Effectiveness of Educational Technology Applications for Enhancing Mathematics Achievement in K-12 Classrooms: A Meta-Analysis. Best Evidence Encyclopedia (BEE). , 2011 .

[94]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[95]  Geoffrey D. Borman.  The Use of Randomized Trials to Inform Education Policy , 2009 .

[96]  R. Kirk.  Practical Significance: A Concept Whose Time Has Come , 1996 .

[97]  Juliana M. Taymans,et al.  Anatomy of an Evidence Base , 2016 .

[98]  Jacobijn Sandberg,et al.  Mobile English learning: An evidence-based study with fifth graders , 2011, Comput. Educ..

[99]  Thomas D. Cook,et al.  Towards a practical theory of external validity , 2000 .

[100]  Ted V. Cascio.  Using Critical Thinking Drills to Teach and Assess Proficiency in Methodological and Statistical Thinking , 2017 .

[101]  Tzu-Chien Liu,et al.  Identifying learning features and models for context-aware ubiquitous learning with phenomenological research method , 2016, Int. J. Mob. Learn. Organisation.

[102]  Il Im,et al.  Comparative Analysis of the Use of Mobile Microblogging and Nonmobile Online Message Board for Group Collaboration , 2015, Int. J. Electron. Commer..

[103]  Selçuk Karaman,et al.  Effects of audience response systems on student achievement and long-term retention , 2011 .

[104]  A. Schoenfeld,et al.  Improving Educational Research:Toward a More Useful, More Influential, and Better-Funded Enterprise , 2003 .

[105]  Yu Zhonggen,et al.  Academic Achievements and Satisfaction of the Clicker-Aided Flipped Business English Writing Class. , 2016 .

[106]  Miguel Nussbaum,et al.  Scaffolding group explanation and feedback with handheld technology: impact on students’ mathematics learning , 2010 .

[107]  J. Michael Spector,et al.  To use augmented reality or not in formative assessment: a comparative study , 2018, Interact. Learn. Environ..

[108]  Chun-Yi Lee,et al.  The effects of image-based concept mapping on the learning outcomes and cognitive processes of mobile learners , 2012, Br. J. Educ. Technol..

[109]  Bryan G. Cook,et al.  CEC’s Standards for Classifying the Evidence Base of Practices in Special Education , 2015 .

[110]  Tzu-Chien Liu,et al.  The effects of integrating mobile devices with teaching and learning on students' learning performance: A meta-analysis and research synthesis , 2016, Comput. Educ..

[111]  Donna L. Sundre,et al.  Seven goals for effective program assessment , 2004 .

[112]  R. Mayer.  The Failure of Educational Research to Impact Educational Practice: Six Obstacles to Educational Reform , 2005 .

[113]  Zhonggen Yu,et al.  Academic Achievements and Satisfaction of the Clicker-Aided Flipped Business English Writing Class , 2016, J. Educ. Technol. Soc..

[114]  R. Pagano.  Understanding Statistics in the Behavioral Sciences , 1981 .

[115]  Nana Yaw Asabere.  The Necessity and Relevance of Mobile Computer-Supported Collaborative Learning (mCSCL): A Review , 2012 .

[116]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[117]  Håkan Fleischer.  What is our current understanding of one-to-one computer projects: A systematic narrative research review , 2012 .