Between Black and White: Examining Grey Literature in Meta-analyses of Psychological Research

The inclusion of grey literature in meta-analyses and reviews is controversial. We examine both the advantages and challenges of including grey literature in meta-analyses. An exemplar meta-analysis of behavioral parenting interventions on parent behavior, child behavior, and parent adjustment outcomes is used to demonstrate these issues. It also explores the influence of the inclusion of grey literature on outcomes, including whether effect sizes are affected by the inclusion of grey literature and describes the challenges of searching for grey literature using traditional search engines such as Google and Yahoo. Homogeneity and publication bias are also examined. Based on these results, recommendations are presented for meta-analysts and researchers.

[1]  H. Eysenck.  The effects of psychotherapy: an evaluation. , 1952, Journal of consulting psychology.

[2]  R. Light,et al.  Accumulating Evidence: Procedures for Resolving Contradictions among Different Research Studies. , 1971 .

[3]  C D Emrick,et al.  A review of psychologically oriented treatment of alcoholism. II. The relative effectiveness of different treatment approaches and the effectiveness of treatment versus no treatment. , 1975, Journal of studies on alcohol.

[4]  M. L. Smith,et al.  Meta-analysis of psychotherapy outcome studies. , 1977, The American psychologist.

[5]  G. Glass.  9: Integrating Findings: The Meta-Analysis of Research , 1977 .

[6]  H. Eysenck.  An exercise in mega-silliness. , 1978 .

[7]  Gene V. Glass,et al.  The benefits of psychotherapy , 1980 .

[8]  A. J. Nassi.  Survivors of the Sixties: Comparative psychosocial and political development of former Berkeley student activists. , 1981 .

[9]  M. L. Klotz,et al.  Effectiveness of psychotherapy with children and adolescents: a meta-analysis for clinicians. , 1987, Journal of consulting and clinical psychology.

[10]  W. McIlroy,et al.  Should unpublished data be included in meta-analyses? Current convictions and controversies. , 1993, JAMA.

[11]  H. Polk,et al.  Peer review is an effective screening process to evaluate medical manuscripts. , 1994, JAMA.

[12]  H S Sacks,et al.  Meta-analysis: an update. , 1996, The Mount Sinai journal of medicine, New York.

[13]  D. Sackett,et al.  Evidence based medicine: what it is and what it isn't , 1996, BMJ.

[14]  J. Dumas,et al.  The effectiveness of behavioral parent training to modify antisocial behavior in children: A meta-analysis , 1996 .

[15]  J. Blythe,et al.  Promoting research utilisation in nursing: the role of the individual, organisation, and environment , 1998 .

[16]  I. Olkin,et al.  Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement , 1999, The Lancet.

[17]  P. Tugwell,et al.  Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? , 2000, The Lancet.

[18]  M. Lipsey,et al.  The role of method in treatment effectiveness research: evidence from meta-analysis. , 2001, Psychological methods.

[19]  M. Clarke,et al.  METHODOLOGISTS AND THEIR METHODS , 2001, International Journal of Technology Assessment in Health Care.

[20]  L. Hedges,et al.  The power of statistical tests in meta-analysis. , 2001, Psychological methods.

[21]  Philip A Scullion,et al.  Effective dissemination strategies. , 2002, Nurse researcher.

[22]  H. Cooper,et al.  Grey Literature in Meta-Analyses , 2003, Nursing research.

[23]  Bryce D. McLeod,et al.  Using dissertations to examine potential bias in child and adolescent clinical trials. , 2004, Journal of consulting and clinical psychology.

[24]  Larry V Hedges,et al.  The power of statistical tests for moderators in meta-analysis. , 2004, Psychological methods.

[25]  E. Decullier,et al.  Fate of biomedical research protocols and publication bias in France: retrospective cohort study , 2005, BMJ : British Medical Journal.

[26]  J. Gilgun.  The Four Cornerstones of Evidence-Based Practice in Social Work , 2005 .

[27]  J. Hamilton.  Clinicians' guide to evidence-based practice. , 2005, Journal of the American Academy of Child and Adolescent Psychiatry.

[28]  W. Jenson,et al.  Behavioral Parent Training as a Treatment for Externalizing Behaviors and Disruptive Behavior Disorders: A Meta-Analysis , 2005 .

[29]  V. Pérez,et al.  Is grey literature essential for a better control of publication bias in psychiatry? An example from three meta-analyses of schizophrenia , 2005, European Psychiatry.

[30]  Brad Lundahl,et al.  A meta-analysis of parent training: moderators and follow-up effects. , 2006, Clinical psychology review.

[31]  R. Levant.  Making psychology a household word. , 2006, The American psychologist.

[32]  Niels Provos,et al.  Search worms , 2006, WORM '06.

[33]  K. A. Hayden,et al.  State-of-the-evidence reviews: advantages and challenges of including grey literature. , 2006, Worldviews on evidence-based nursing.

[34]  R. Azen,et al.  Differential Effectiveness of Behavioral Parent-Training and Cognitive-Behavioral Therapy for Antisocial Youth: A Meta-Analysis , 2006, Journal of abnormal child psychology.

[35]  How to search for and use 'grey literature' in research. , 2006, Nursing times.

[36]  P. McGrath,et al.  Predictors of parent training efficacy for child externalizing behavior problems--a meta-analytic review. , 2006, Journal of child psychology and psychiatry, and allied disciplines.

[37]  A. Beck,et al.  The empirical status of cognitive-behavioral therapy: a review of meta-analyses. , 2006, Clinical psychology review.

[38]  M Egger,et al.  Grey literature in meta-analyses of randomized trials of health care interventions. , 2007, The Cochrane database of systematic reviews.

[39]  Searching for and using grey literature , 2007 .

[40]  M. Zimmer‐Gembeck,et al.  Behavioral Outcomes of Parent-Child Interaction Therapy and Triple P—Positive Parenting Program: A Review and Meta-Analysis , 2007, Journal of abnormal child psychology.

[41]  O. Bukstein,et al.  American Academy of Child and Adolescent Psychiatry , 2018, The Grants Register 2022.

[42]  Elizabeth Taylor,et al.  What Would You Tell Me If I Said Grey Literature? , 2007 .

[43]  John Andy Wood,et al.  Methodology for Dealing With Duplicate Study Effects in a Meta-Analysis , 2008 .

[44]  John Hunsley,et al.  A Guide to Assessments That Work , 2008 .

[45]  Wesley G. Jennings,et al.  Effects of early family/parent training programs on antisocial behavior and delinquency , 2008 .

[46]  Jennifer Wyatt Kaminski,et al.  A Meta-analytic Review of Components Associated with Parent Training Program Effectiveness , 2008, Journal of abnormal child psychology.

[47]  Karen A. Blase,et al.  Research utilization among children's mental health providers , 2008, Implementation science : IS.

[48]  N. Heinrichs,et al.  A Comprehensive Meta-Analysis of Triple P-Positive Parenting Program Using Hierarchical Linear Modeling: Effectiveness and Moderating Variables , 2008, Clinical child and family psychology review.

[49]  Derrik R. Tollefson,et al.  A Meta-Analysis of Father Involvement in Parent Training , 2008 .

[50]  F. Smit,et al.  UvA-DARE ( Digital Academic Repository ) Effectiveness of the Triple P Positive Parenting Program on Parenting : A meta-analysis , 2016 .

[51]  J. Ioannidis,et al.  Systematic Review of the Empirical Evidence of Study Publication Bias and Outcome Reporting Bias , 2008, PloS one.

[52]  J. Hunsley,et al.  Developing criteria for evidence-based assessment: An introduction to assessments that work. , 2008 .

[53]  E. Parens,et al.  Facts, values, and Attention-Deficit Hyperactivity Disorder (ADHD): an update on the controversies , 2009, Child and Adolescent Psychiatry and Mental Health.

[54]  S. Stewart-Brown,et al.  The clinical effectiveness of different parenting programmes for children with conduct problems : a systematic review of randomised controlled trials , 2015 .

[55]  Marcus Banks.  Blog posts and tweets: the next frontier for grey literature , 2009 .

[56]  D. Moher,et al.  CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials , 2010, Journal of clinical epidemiology.

[57]  D. Moher,et al.  CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials , 2010, BMJ : British Medical Journal.

[58]  Joachim Schöpfel,et al.  Grey Literature in Library and Information Studies , 2010 .

[59]  Barbara J. Roth.  An Academic Perspective on Grey Literature , 2010 .

[60]  Deni J. Seymour.  Sanctioned Inequity and Accessibility Issues in the Grey Literature in the United States , 2010 .

[61]  S. Matthey,et al.  The impact of behavioural parent training on fathers' parenting: a meta-analysis of the Triple-P Positive Parenting Program , 2011 .

[62]  Cleo Pappas,et al.  Grey Literature: Its Emerging Importance , 2011 .

[63]  Kate E Decleene,et al.  Publication Manual of the American Psychological Association , 2011 .

[64]  G. Mays,et al.  Expanding delivery system research in public health settings: lessons from practice-based research networks. , 2012, Journal of public health management and practice : JPHMP.

[65]  S. McGilloway,et al.  Behavioural and cognitive-behavioural group-based parenting programmes for early-onset conduct problems in children aged 3 to 12 years. , 2012, The Cochrane database of systematic reviews.

[66]  Sarah P. Bellefontaine.  A Meta-Analytic Examination of Behavioural Parenting Interventions in the Treatment of Children's Behaviour Problems , 2013 .

[67]  P. Mahar,et al.  Intravenous lidocaine for the treatment of background or procedural burn pain. , 2014, The Cochrane database of systematic reviews.