The overview of reviews: unique challenges and opportunities when research syntheses are the principal elements of new integrative scholarship.

In the past two decades, a new form of scholarship has appeared in which researchers present an overview of previously conducted research syntheses on the same topic. In these efforts, research syntheses are the principal units of evidence. Overviews of reviews introduce unique problems that require unique solutions. This article describes what methods overviewers have developed or have adopted from other forms of scholarship. These methods concern how to (a) define the broader problem space of an overview, (b) conduct literature searches that specifically look for research syntheses, (c) address the overlap in evidence in related reviews, (d) evaluate the quality of both primary research and research syntheses, (e) integrate the outcomes of research syntheses, especially when they produce discordant results, (f) conduct a second-order meta-analysis, and (g) present findings. The limitations of overviews are also discussed, especially with regard to the age of the included evidence.

[1]  David Rindskopf,et al.  APA handbook of research methods in psychology, Vol 3: Data analysis and research publication. , 2012 .

[2]  Vivian C. Wong,et al.  A randomized experiment comparing random and cutoff-based assignment. , 2011, Psychological methods.

[3]  Philip C. Abrami,et al.  What Forty Years of Research Says About the Impact of Technology on Learning , 2011 .

[4]  M. Clarke,et al.  Methodology in conducting a systematic review of systematic reviews of healthcare interventions , 2011, BMC medical research methodology.

[5]  What Works Clearinghouse: Procedures and Standards Handbook (Version 2.1) , 2011 .

[6]  Stephen Platt,et al.  The effectiveness of interventions to change six health behaviours: a review of reviews , 2010, BMC public health.

[7]  R. Maniglio,et al.  Child sexual abuse in the etiology of depression: A systematic review of reviews , 2010, Depression and anxiety.

[8]  Denise Thomson,et al.  The evolution of a new publication type: Steps and challenges of producing overviews of reviews , 2010, Research synthesis methods.

[9]  Harris Cooper,et al.  A method for evaluating research syntheses: The quality, conclusions, and consensus of 12 syntheses of the effects of after‐school programs , 2010, Research synthesis methods.

[10]  L. Hedges,et al.  The Handbook of Research Synthesis and Meta-Analysis , 2009 .

[11]  John P.A. Ioannidis,et al.  Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses , 2009, Canadian Medical Association Journal.

[12]  Jeremy Grimshaw,et al.  AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. , 2009, Journal of clinical epidemiology.

[13]  David Moher,et al.  An evidence-based practice guideline for the peer review of electronic search strategies. , 2009, Journal of clinical epidemiology.

[14]  Gerjo Kok,et al.  Effective elements of school health promotion across behavioral domains: a systematic review of reviews , 2009, BMC public health.

[15]  Harris Cooper,et al.  Research Synthesis and Meta-Analysis: A Step-by-Step Approach , 2009 .

[16]  A. Butchart,et al.  Child maltreatment prevention: a systematic review of reviews. , 2009, Bulletin of the World Health Organization.

[17]  Dorothy Newbury-Birch,et al.  Impact of alcohol consumption on young people : a review of reviews , 2009 .

[18]  Apa Publications,et al.  Reporting standards for research in psychology: why do we need them? What might they be? , 2008, The American psychologist.

[19]  A. Oxman,et al.  Overviews of Reviews , 2008 .

[20]  D. Altman,et al.  Assessing Risk of Bias in Included Studies , 2008 .

[21]  Harris Cooper,et al.  A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: the Study Design and Implementation Assessment Device (Study DIAD). , 2008, Psychological methods.

[22]  Jan Brozek,et al.  GRADE: assessing the quality of evidence for diagnostic recommendations , 2008, Annals of Internal Medicine.

[23]  David Moher,et al.  Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews , 2007, BMC medical research methodology.

[24]  Harris Cooper,et al.  Evaluating and Interpreting Research Syntheses in Adult Learning and Literacy. NCSALL Occasional Paper. , 2007 .

[25]  J. Coyne,et al.  Psychological interventions for distress in cancer patients: a review of reviews , 2006, Annals of behavioral medicine : a publication of the Society of Behavioral Medicine.

[26]  John Cowan,et al.  The quality of research , 2006, Br. J. Educ. Technol..

[27]  R Brian Haynes,et al.  Developing optimal search strategies for detecting clinically sound treatment studies in EMBASE. , 2006, Journal of the Medical Library Association : JMLA.

[28]  A. Beck,et al.  The empirical status of cognitive-behavioral therapy: a review of meta-analyses. , 2006, Clinical psychology review.

[29]  E. Waters,et al.  Promoting the Social and Emotional Health of Primary School-Aged Children: Reviewing the Evidence Base for School - Based Interventions , 2005 .

[30]  P. Petersen.  The burden of oral disease: challenges to improving oral health in the 21st century. , 2005, Bulletin of the World Health Organization.

[31]  R. Haynes,et al.  Optimal search strategies for retrieving systematic reviews from Medline: analytical survey , 2004, BMJ : British Medical Journal.

[32]  B. Naidoo.  Smoking and Public Health: A Review of Reviews of Interventions to Increase Smoking Cessation, Reduce Smoking Initiation and Prevent Further Uptake of Smoking - Evidence Briefing , 2004 .

[33]  Cindy A. Crusto,et al.  What works in prevention. Principles of effective prevention programs. , 2003, The American psychologist.

[34]  N McKoy,et al.  Systems to rate the strength of scientific evidence. , 2002, Evidence report/technology assessment.

[35]  M. Lipsey,et al.  The role of method in treatment effectiveness research: evidence from meta-analysis. , 2001, Psychological methods.

[36]  R. Peterson.  On the Use of College Students in Social Science Research: Insights from a Second‐Order Meta‐analysis , 2001 .

[37]  Katherine Irwin,et al.  Blueprints for Violence Prevention , 2001 .

[38]  Murray R. Barrick,et al.  Personality and Performance at the Beginning of the New Millennium: What Do We Know and Where Do We Go Next? , 2001 .

[39]  Ihab S. Soliman,et al.  Stimulant medications for the treatment of ADHD: Efficacy and limitations , 1999 .

[40]  D. Cook,et al.  A guide to interpreting discordant systematic reviews. , 1997, CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne.

[41]  Mark W. Lipsey,et al.  The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. , 1993, The American psychologist.

[42]  Francis M. Crinella,et al.  Effect of Stimulant Medication on Children with Attention Deficit Disorder: A “Review of Reviews” , 1993 .

[43]  J T Grace.  Quality of research. , 1990, Journal of obstetric, gynecologic, and neonatal nursing : JOGNN.

[44]  Frank L. Schmidt,et al.  Biographical Data in Employment Selection: Can Validities Be Made Generalizable? , 1990 .

[45]  H. Cooper.  Organizing knowledge syntheses: A taxonomy of literature reviews , 1988 .

[46]  A. Kazrin,et al.  Meta-meta analysis: a new method for evaluating therapy outcome. , 1979, Behaviour research and therapy.

[47]  D J PRICE,et al.  NETWORKS OF SCIENTIFIC PAPERS. , 1965, Science.