An exploration of bias in meta-analysis: the case of technology integration research in higher education

This article contains a second-order meta-analysis and an exploration of bias in the technology integration literature in higher education. Thirteen meta-analyses, dated from 2000 to 2014 were selected to be included based on the questions asked and the presence of adequate statistical information to conduct a quantitative synthesis. The weighted random effects average was g++ = 0.393, p < .000. The article goes on to report an assessment of the methodological quality of the thirteen studies based on Cooper’s (Research synthesis and meta-analysis: a step-by-step approach. Sage, Thousand Oaks, 2010) seven stages in the development of a meta-analysis. Two meta-analyses were found to have five out of seven stages where methodological flaws could potentially create biased results. Five meta-analyses contained two flawed stages and one contained one flawed stage. Four of the stages where methodological flaws can create bias are described in detail. The final section attempts to determine how much influence the methodological flaws exerted on the results of the second-order meta-analysis.

[1]  David Moher,et al.  Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews , 2007, BMC medical research methodology.

[2]  Philip C. Abrami,et al.  What Forty Years of Research Says About the Impact of Technology on Learning , 2011 .

[3]  Michelle F. Wright,et al.  Computer Gaming and Interactive Simulations for Learning: A Meta-Analysis , 2006 .

[4]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[5]  John Hattie,et al.  Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement , 2008 .

[6]  Ahmet Tekbıyık,et al.  A Meta-Analytical Investigation of the Influence of Computer Assisted Instruction on Achievement in Science , 2010 .

[7]  Karen H. Larwin,et al.  A Meta-Analysis Examining the Impact of Computer-Assisted Instruction on Postsecondary Statistics Education , 2011 .

[8]  D. Cook.  The failure of e-learning research to inform educational practice, and what we can do about it , 2009, Medical teacher.

[9]  Vivien E. Rolfe,et al.  Are Multimedia Resources Effective in Life Science Education? A Meta-Analysis , 2011 .

[10]  Sally Hollis,et al.  A tool to assess the quality of a meta‐analysis , 2013, Research synthesis methods.

[11]  Ralf W. Schlosser,et al.  Searching for evidence in augmentative and alternative communication: Navigating a scattered literature , 2005 .

[12]  Gregg B. Jackson,et al.  Methods for Integrative Reviews , 1980 .

[13]  Robert M. Bernard,et al.  A Meta-Analysis of Three Types of Interaction Treatments in Distance Education , 2009 .

[14]  Harris Cooper,et al.  The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship. , 2012, The American psychologist.

[15]  Harris Cooper,et al.  Research Synthesis and Meta-Analysis: A Step-by-Step Approach , 2009 .

[16]  Richard F. Schmid,et al.  The effects of technology use in postsecondary education: A meta-analysis of classroom applications , 2014, Comput. Educ..

[17]  S Duval,et al.  Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis , 2000, Biometrics.

[18]  L. Hedges,et al.  Introduction to Meta‐Analysis , 2009, International Coaching Psychology Review.

[19]  Robert M. Bernard.  Things I Have Learned about Meta-­Analysis since 1990: Reducing Bias in Search of "The Big Picture" , 2014 .

[20]  Hannah R Rothstein,et al.  A basic introduction to fixed‐effect and random‐effects models for meta‐analysis , 2010, Research synthesis methods.

[21]  M. Borenstein,et al.  Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments , 2006 .

[22]  Matthew K. Burns,et al.  Updated Meta-Analysis of Learner Control Within Educational Technology , 2014 .

[23]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[24]  Wolfgang Viechtbauer,et al.  Outlier and influence diagnostics for meta‐analysis , 2010, Research synthesis methods.

[25]  John E. Hunter,et al.  Methods of Meta-Analysis: Correcting Error and Bias in Research Findings , 1991 .

[26]  Traci Sitzmann.  A META-ANALYTIC EXAMINATION OF THE INSTRUCTIONAL EFFECTIVENESS OF COMPUTER-BASED SIMULATION GAMES , 2011 .

[27]  Andreas Rosenblad,et al.  Introduction to Meta‐Analysis by Michael Borenstein, Larry V. Hedges, Julian P.T. Higgins, Hannah R. Rothstein , 2009 .

[28]  G. Glass.  Primary, Secondary, and Meta-Analysis of Research1 , 1976 .

[29]  Zahira Merchant,et al.  Effectiveness of virtual reality-based instruction on students' learning outcomes in K-12 and higher education: A meta-analysis , 2014, Comput. Educ..

[30]  N. Wiebe,et al.  Effective Methods for Teaching Information Literacy Skills to Undergraduate Students: A Systematic Review and Meta-analysis , 2006 .

[31]  Mark W. Lipsey,et al.  Practical Meta-Analysis , 2000 .

[32]  Yung-Chen Hsu,et al.  The effectiveness of computer-assisted instruction in statistics education: a meta-analysis , 2003 .

[33]  Philip C. Abrami,et al.  Issues in conducting and disseminating brief reviews of evidence , 2010 .

[34]  Greg Roberts,et al.  Meta-Analysis With Complex Research Designs , 2014, Review of educational research.

[35]  C. Erik Timmerman,et al.  Computer-Assisted Instruction, Media Richness, and College Student Performance , 2006 .

[36]  Yong Zhao,et al.  Recent Developments in Technology and Language Learning: A Literature Review and Meta-analysis , 2013 .

[37]  Dale E. Berger,et al.  Effectiveness of Computer-Assisted Instruction in Statistics , 2011 .

[38]  Harris Cooper,et al.  A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: the Study Design and Implementation Assessment Device (Study DIAD). , 2008, Psychological methods.

[39]  Edwin P. Christmann,et al.  The comparative effectiveness of CAI on collegiate academic performance , 2000, J. Comput. High. Educ..

[40]  Richard F. Schmid,et al.  A meta-analysis of blended learning and technology use in higher education: from the general to the applied , 2014, Journal of Computing in Higher Education.