Testing for publication bias in diagnostic meta‐analysis: a simulation study

The present study investigates the performance of several statistical tests to detect publication bias in diagnostic meta-analysis by means of simulation. While bivariate models should be used to pool data from primary studies in diagnostic meta-analysis, univariate measures of diagnostic accuracy are preferable for the purpose of detecting publication bias. In contrast to earlier research, which focused solely on the diagnostic odds ratio or its logarithm ( ln ω), the tests are combined with four different univariate measures of diagnostic accuracy. For each combination of test and univariate measure, both type I error rate and statistical power are examined under diverse conditions. The results indicate that tests based on linear regression or rank correlation cannot be recommended in diagnostic meta-analysis, because type I error rates are either inflated or power is too low, irrespective of the applied univariate measure. In contrast, the combination of trim and fill and ln ω has non-inflated or only slightly inflated type I error rates and medium to high power, even under extreme circumstances (at least when the number of studies per meta-analysis is large enough). Therefore, we recommend the application of trim and fill combined with ln ω to detect funnel plot asymmetry in diagnostic meta-analysis.

[1]  Chen-Tuo Liao,et al.  A non‐inferiority test for diagnostic accuracy based on the paired partial areas under ROC curves , 2008, Statistics in medicine.

[2]  W. Youden,et al.  Index for rating diagnostic tests , 1950, Cancer.

[3]  Roger M Harbord,et al.  A unification of models for meta-analysis of diagnostic accuracy studies. , 2007, Biostatistics.

[4]  C. Begg,et al.  Operating characteristics of a rank correlation test for publication bias. , 1994, Biometrics.

[5]  Colin B. Begg,et al.  An Approach for Assessing Publication Bias Prior to Performing a Meta-Analysis , 1992 .

[6]  Roger M Harbord,et al.  A modified test for small‐study effects in meta‐analyses of controlled trials with binary endpoints , 2006, Statistics in medicine.

[7]  Chap T Le,et al.  A solution for the most basic optimization problem associated with an ROC curve , 2006, Statistical methods in medical research.

[8]  R. Tweedie,et al.  Publication Bias in Meta-Analysis: A Bayesian Data-Augmentation Approach to Account for Issues Exempli(cid:12)ed in the Passive Smoking Debate , 1997 .

[9]  P. Easterbrook,et al.  Publication bias in clinical research , 1991, The Lancet.

[10]  J. Sterne,et al.  Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. , 2000, Journal of clinical epidemiology.

[11]  William R. Shadish,et al.  Using odds ratios as effect sizes for meta-analysis of dichotomous data: A primer on methods and issues. , 1998 .

[12]  L. Hedges.  Modeling publication selection effects in meta-analysis , 1992 .

[13]  Jen‐pei Liu,et al.  Tests of equivalence and non‐inferiority for diagnostic accuracy based on the paired areas under ROC curves , 2006, Statistics in medicine.

[14]  Patrick Bossuyt,et al.  Systematic Reviews of Diagnostic Test Accuracy , 2008, Annals of Internal Medicine.

[15]  J. Fleiss.  Measuring nominal scale agreement among many raters. , 1971 .

[16]  L E Moses,et al.  Estimating Diagnostic Accuracy from Multiple Conflicting Reports , 1993, Medical decision making : an international journal of the Society for Medical Decision Making.

[17]  S Duval,et al.  Trim and Fill: A Simple Funnel‐Plot–Based Method of Testing and Adjusting for Publication Bias in Meta‐Analysis , 2000, Biometrics.

[18]  B. Everitt,et al.  Large sample standard errors of kappa and weighted kappa. , 1969 .

[19]  Dirk Eddelbuettel,et al.  Rcpp: Seamless R and C++ Integration , 2011 .

[20]  D. Bloch,et al.  Comparing two diagnostic tests against the same "gold standard" in the same sample. , 1997, Biometrics.

[21]  Gerta Rücker,et al.  Arcsine test for publication bias in meta‐analyses with binary outcomes , 2008, Statistics in medicine.

[22]  Christopher H Schmid,et al.  In an empirical evaluation of the funnel plot, researchers could not visually identify publication bias. , 2005, Journal of clinical epidemiology.

[23]  Guido Schwarzer,et al.  A test for publication bias in meta‐analysis with sparse binary data , 2007, Statistics in medicine.

[24]  S. Sharp,et al.  Explaining heterogeneity in meta-analysis: a comparison of methods. , 1999 .

[25]  Heinz Holling,et al.  Meta-analysis of diagnostic studies based upon SROC-curves: a mixed model approach using the Lehmann family , 2012 .

[26]  Frederick Mosteller,et al.  Guidelines for Meta-analyses Evaluating Diagnostic Tests , 1994, Annals of Internal Medicine.

[27]  C. Begg,et al.  Publication bias : a problem in interpreting medical data , 1988 .

[28]  Jonathan J Deeks,et al.  The performance of tests of publication bias and other sample size effects in systematic reviews of diagnostic test accuracy was assessed. , 2005, Journal of clinical epidemiology.

[29]  C M Rutter,et al.  A hierarchical regression approach to meta‐analysis of diagnostic test accuracy evaluations , 2001, Statistics in medicine.

[30]  Guido Schwarzer,et al.  Inflation of type I error rate in two statistical tests for the detection of publication bias in meta‐analyses with binary outcomes , 2002, Statistics in medicine.

[31]  E. Tacconelli.  Systematic reviews: CRD's guidance for undertaking reviews in health care , 2010 .

[32]  Art Noda,et al.  Kappa coefficients in medical research , 2002, Statistics in medicine.

[33]  Johannes B Reitsma,et al.  Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. , 2005, Journal of clinical epidemiology.

[34]  Heinz Holling,et al.  Likelihood-Based Clustering of Meta-Analytic SROC Curves , 2012 .

[35]  Alexander J Sutton,et al.  What to add to nothing? Use and avoidance of continuity corrections in meta-analysis of sparse data. , 2004, Statistics in medicine.

[36]  Alex J Sutton,et al.  Performance of the trim and fill method in the presence of publication bias and between‐study heterogeneity , 2007, Statistics in medicine.

[37]  James D. Stamey,et al.  A Bayesian approach to sample size determination for studies designed to evaluate continuous medical tests , 2010, Comput. Stat. Data Anal..

[38]  J. C. Houwelingen,et al.  Bivariate Random Effects Meta-Analysis of ROC Curves , 2008, Medical decision making : an international journal of the Society for Medical Decision Making.

[39]  Heinz Holling,et al.  Revisiting youden's index as a useful measure of the misclassification error in meta-analysis of diagnostic studies , 2008, Statistical methods in medical research.

[40]  David Moher,et al.  Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative. , 2003, Radiology.

[41]  Heinz Holling,et al.  A limitation of the diagnostic-odds ratio in determining an optimal cut-off value for a continuous diagnostic test , 2011, Statistical methods in medical research.

[42]  A. Sutton,et al.  Comparison of two methods to detect publication bias in meta-analysis. , 2006, JAMA.

[43]  Haitao Chu,et al.  Statistical methods for multivariate meta-analysis of diagnostic tests: An overview and tutorial , 2016, Statistical methods in medical research.

[44]  P Glasziou,et al.  Meta-analytic methods for diagnostic test accuracy. , 1995, Journal of clinical epidemiology.

[45]  M. Kendall.  A NEW MEASURE OF RANK CORRELATION , 1938 .

[46]  Alex J Sutton,et al.  Asymmetric funnel plots and publication bias in meta-analyses of diagnostic accuracy. , 2002, International journal of epidemiology.

[47]  S D Walter,et al.  A comparison of methods to detect publication bias in meta‐analysis , 2001, Statistics in medicine.

[48]  P. Bossuyt,et al.  The diagnostic odds ratio: a single indicator of test performance. , 2003, Journal of clinical epidemiology.