What Is the State of theArt of Systematic Reviewin Engineering Education?

Background Systematic review refers to a growing set of research methodologies intended to critically appraise and synthesize research to inform policy and practice. Systematic review methods, widely used in fields such as medicine, psychology, and education, are being applied in engineering education. Analysis of current systematic review practices in engineering education can contribute to continuous improvement of an important set of synthesis methodologies. Purpose To promote continuous improvement in systematic reviews, researchers are currently developing criteria to evaluate the quality of systematic review methodologies. The purpose of this article is to identify prior systematic reviews in areas of interest to the engineering education community, use these criteria to critique these systematic reviews, and make recommendations for improving use and quality of systematic review methods in engineering education research. Scope/Method Using systematic review search methodologies, we identified 12 conference papers and 37 journal articles that are systematic reviews published since 1990. Two reviewers coded each by topic, education level, quantitative or qualitative sources and synthesis methods, and 18 quantitative measures of review quality. Interrater reliability was calculated using kappa as described by Landis and Koch, which ranged from 0.43 (moderate) to 0.80 (almost perfect) with overall agreement of 0.64 (substantial). Conclusions We found that the 49 identified systematic reviews in engineering education are particularly strong in stating their purposes, rationale, and inclusion criteria. They can be improved by explaining reasons for excluding studies, appraising quality of included studies, engaging multiple coders in evaluation procedures, stating their limitations, and, in the case of meta-analyses, describing statistical methods in greater detail.

[1]  P. Wankat.  An Analysis of the Articles in the Journal of Engineering Education , 1999 .

[2]  D. Finfgeld,et al.  Metasynthesis: The State of the Art—So Far , 2003, Qualitative health research.

[3]  Mario Piattini,et al.  Preparing Students and Engineers for Global Software Development: A Systematic Review , 2010, 2010 5th IEEE International Conference on Global Software Engineering.

[4]  Billy L. Crynes,et al.  A Review of Literature on Effectiveness of Use of Information Technology in Education , 2000 .

[5]  Jeffrey V. Nickerson,et al.  Hands-on, simulated, and remote laboratories: A comparative literature review , 2006, CSUR.

[6]  Lynne Jump,et al.  Why university lecturers enhance their teaching through the use of technology: a systematic review , 2011 .

[7]  David A Cook,et al.  Conducting systematic reviews in medical education: a stepwise approach , 2012, Medical education.

[8]  Matthew W. Ohland,et al.  Creating A Catalog And Meta Analysis Of Freshman Programs For Engineering Students: Part 2: Learning Communities , 2002 .

[9]  E. Douglas,et al.  State of Qualitative Research in Engineering Education: Meta‐Analysis of JEE Articles, 2005–2006 , 2008 .

[10]  Fabiane Barreto Vavassori Benitti,et al.  Exploring the educational potential of robotics in schools: A systematic review , 2012, Comput. Educ..

[11]  Philip C. Abrami,et al.  What Forty Years of Research Says About the Impact of Technology on Learning , 2011 .

[12]  Krishna Madhavan,et al.  Anatomy of Scholarly Collaboration in Engineering Education: A Big‐Data Bibliometric Analysis , 2014 .

[13]  Matthew W. Ohland,et al.  Creating A Catalog And Meta Analysis Of Freshman Programs For Engineering Students: Part 1: Summer Bridge Programs , 2002 .

[14]  David S. Cordray,et al.  A Research Synthesis of the Effectiveness, Replicability, and Generality of the VaNTH Challenge‐based Instructional Modules in Bioengineering , 2009 .

[15]  David W. Johnson,et al.  Cooperative Learning Returns To College What Evidence Is There That It Works , 1998 .

[16]  Maura Borrego,et al.  Mapping global trends in engineering education research , 2011 .

[17]  Amy Orange,et al.  An Evaluation of HigherEd 2.0 Technologies in Undergraduate Mechanical Engineering Courses. , 2012 .

[18]  P. Fensham.  Defining an Identity , 2004 .

[19]  Daniel J. Fonseca,et al.  A Meta Analysis Of Studies Of Cognition In Engineering Education , 2010 .

[20]  Ian T. Cameron,et al.  Development and deployment of a library of industrially focused advanced immersive VR learning environments , 2008 .

[21]  J. W. Gikandi,et al.  Online formative assessment in higher education: A review of the literature , 2011, Comput. Educ..

[22]  D. Uttal,et al.  The malleability of spatial skills: a meta-analysis of training studies. , 2013, Psychological bulletin.

[23]  Craig E. Kuziemsky,et al.  A review on systematic reviews of health information system studies , 2010, J. Am. Medical Informatics Assoc..

[24]  Venkataraman Ramesh,et al.  An analysis of research in computing disciplines , 2004, CACM.

[25]  Hude Quan,et al.  An administrative data merging solution for dealing with missing data in a clinical registry: adaptation from ICD-9 to ICD-10 , 2008, BMC medical research methodology.

[26]  S. M. Gómez Puente,et al.  Towards characterising design-based learning in engineering education: a review of the literature , 2011 .

[27]  C. Henderson,et al.  Facilitating change in undergraduate STEM instructional practices: An analytic review of the literature , 2011 .

[28]  Qing Li,et al.  A Meta-analysis of the Effects of Computer Technology on School Students’ Mathematics Learning , 2010 .

[29]  Kathrin Figl,et al.  A Systemic Review of Developing Team Competencies in Information Systems Education , 2010, J. Inf. Syst. Educ..

[30]  K. Knobloch,et al.  Preferred reporting items for systematic reviews and meta-analyses (PRISMA) statement and publication bias. , 2011, Journal of cranio-maxillo-facial surgery : official publication of the European Association for Cranio-Maxillo-Facial Surgery.

[31]  Phillip C. Wankat,et al.  Analysis of the First Ten Years of the Journal of Engineering Education , 2004 .

[32]  J. Froyd,et al.  Systematic Literature Reviews in Engineering Education and Other Developing Interdisciplinary Fields , 2014 .

[33]  Christopher Papadopoulos,et al.  Do Students In Summer Bridge Programs Successfully Improve Math Placement And Persist? A Meta Analysis. , 2008 .

[34]  Michael J. Prince,et al.  Does Active Learning Work? A Review of the Research , 2004 .

[35]  C. Henderson,et al.  Beyond Dissemination in College Science Teaching: An Introduction to Four Core Change Strategies , 2010 .

[36]  M. Kadiyala,et al.  Where's the proof? A review of literature on effectiveness of information technology in education , 1998, FIE '98. 28th Annual Frontiers in Education Conference. Moving from 'Teacher-Centered' to 'Learner-Centered' Education. Conference Proceedings (Cat. No.98CH36214).

[37]  James F. Power,et al.  Teaching discrete structures: a systematic review of the literature , 2011, SIGCSE '11.

[38]  Craig Caulfield,et al.  A systematic survey of games used for software engineering education , 2011 .

[39]  N. Falchikov,et al.  Student Peer Assessment in Higher Education: A Meta-Analysis Comparing Peer and Teacher Marks , 2000 .

[40]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[41]  Maura Borrego,et al.  Development of Engineering Education as a Rigorous Discipline: A Study of the Publication Patterns of Four Coalitions , 2007 .

[42]  James M. Boyle,et al.  A systematic literature review of empirical evidence on computer games and serious games , 2012, Comput. Educ..

[43]  Noah D. Finkelstein,et al.  Facilitating Change in Undergraduate STEM: Initial Results from an Interdisciplinary Literature Review , 2008 .

[44]  S. Sheppard,et al.  Taking Stock: An Analysis of the Publishing Record as Represented by the Journal of Engineering Education , 2004 .

[45]  Maura Borrego,et al.  Identifying Opportunities for Collaborations in International Engineering Education Research on Problem- and Project-based Learning , 2010 .

[46]  Raymond Mugwanya,et al.  Mobile Learning Content Authoring Tools (MLCATs): A Systematic Review , 2009, AFRICOM.

[47]  D. Haws.  Ethics Instruction in Engineering Education: A (Mini) Meta‐Analysis , 2001 .

[48]  André L. M. Santos,et al.  A critical appraisal of systematic reviews in software engineering from the perspective of the research questions asked in the reviews , 2010, ESEM '10.

[49]  Robin Adams,et al.  The Informed Design Teaching and Learning Matrix , 2012 .

[50]  Johan van Braak,et al.  Preparing pre-service teachers to integrate technology in education: A synthesis of qualitative evidence , 2012, Comput. Educ..

[51]  Marjan Hericko,et al.  A meta-analysis of e-learning technology acceptance: The role of user types and e-learning technology types , 2011, Comput. Hum. Behav..

[52]  Anastasios Karakostas,et al.  Adaptive and Intelligent Systems for Collaborative Learning Support: A Review of the Field , 2011, IEEE Transactions on Learning Technologies.

[53]  E Ernst,et al.  A systematic review of systematic reviews of homeopathy. , 2002, British journal of clinical pharmacology.

[54]  Jennifer Karlin,et al.  Effective Faculty Mentoring For Diversity: An Assessment Of Mentoring Paradigms , 2010 .

[55]  Kurt Becker,et al.  Integrative Approaches among Science,Technology, Engineering and Mathematics (STEM) Subjects on Students' Learning: AMeta-Analysis , 2009 .

[56]  Aditya Johri,et al.  Creating Theoretical Insights in Engineering Education , 2010 .

[57]  G.M. Michko,et al.  Meta-analysis of effectiveness of technology use in undergraduate engineering education , 2008, 2008 38th Annual Frontiers in Education Conference.

[58]  Leonard Springer,et al.  Effects of Small-Group Learning on Undergraduates in Science, Mathematics, Engineering, and Technology: A Meta-Analysis , 1997 .

[59]  V. Wilson,et al.  Designing the Best: A Review of Effective Teaching and Learning of Design and Technology , 2003 .

[60]  C. W. Bowen,et al.  A Quantitative Literature Review of Cooperative Learning Effects on High School and College Chemistry Achievement. , 2000 .

[61]  Theocharis Kyriacou,et al.  Systematic literature review: Teaching novices programming using robots , 2011 .

[62]  Pearl Brereton,et al.  Systematic literature reviews in software engineering - A systematic literature review , 2009, Inf. Softw. Technol..

[63]  M. Borrego,et al.  Feminist Theory in Three Engineering Education Journals: 1995–2008 , 2011 .

[64]  Lauri Malmi,et al.  Characterizing research in computing education: a preliminary analysis of the literature , 2010, ICER '10.

[65]  James E. Folkestad,et al.  STEM Education Related Dissertation Abstracts: A Bounded Qualitative Meta-study , 2012 .

[66]  R Ruurd Taconis,et al.  Teaching Science Problem Solving: An Overview of Experimental Work. , 2001 .

[67]  So Yoon Yoon,et al.  A Meta-Analysis on Gender Differences in Mental Rotation Ability Measured by the Purdue Spatial Visualization Tests: Visualization of Rotations (PSVT:R) , 2013 .

[68]  Jing Wang,et al.  Conceptualizing Authenticity in Engineering Education: A Systematic Literature Review , 2012 .

[69]  David Moher,et al.  The art and science of knowledge synthesis. , 2011, Journal of clinical epidemiology.

[70]  Richard F. Schmid,et al.  Technology’s effect on achievement in higher education: a Stage I meta-analysis of classroom applications , 2009, J. Comput. High. Educ..

[71]  Johan van Braak,et al.  Supporting self-regulated learning in computer-based learning environments: systematic review of effects of scaffolding in the domain of science education , 2012, J. Comput. Assist. Learn..

[72]  G. Salomon,et al.  The Differential Learning Achievements of Constructivist Technology-Intensive Learning Environments as Compared with Traditional Ones: A Meta-Analysis , 2007 .

[73]  Peter C Gøtzsche,et al.  ' s response to reviews Title : Spontaneous improvement in randomised clinical trials : meta-analysis of three-armed trials comparing no-treatment , placebo and active intervention , 2008 .

[74]  Nadia Kellam,et al.  Quality in Interpretive Engineering Education Research: Reflections on an Example Study , 2013 .

[75]  Emilia Mendes,et al.  Empirical Studies of Pair Programming for CS/SE Teaching in Higher Education: A Systematic Literature Review , 2011, IEEE Transactions on Software Engineering.

[76]  Cathy Gunn,et al.  A portrait of evaluation studies of learning technology innovations 2005-2010: Addressing the elephant in the room , 2011 .

[77]  Y. Lou,et al.  Understanding Process and Affective Factors in Small Group versus Individual Learning with Technology , 2004 .

[78]  Chris McMahon,et al.  Knowledge sharing for sustainable development in civil engineering: a systematic review , 2011, AI & SOCIETY.