Newcastle-Ottawa Scale: comparing reviewers’ to authors’ assessments

BackgroundLack of appropriate reporting of methodological details has previously been shown to distort risk of bias assessments in randomized controlled trials. The same might be true for observational studies. The goal of this study was to compare the Newcastle-Ottawa Scale (NOS) assessment for risk of bias between reviewers and authors of cohort studies included in a published systematic review on risk factors for severe outcomes in patients infected with influenza.MethodsCohort studies included in the systematic review and published between 2008–2011 were included. The corresponding or first authors completed a survey covering all NOS items. Results were compared with the NOS assessment applied by reviewers of the systematic review. Inter-rater reliability was calculated using kappa (K) statistics.ResultsAuthors of 65/182 (36%) studies completed the survey. The overall NOS score was significantly higher (p < 0.001) in the reviewers’ assessment (median = 6; interquartile range [IQR] 6–6) compared with those by authors (median = 5, IQR 4–6). Inter-rater reliability by item ranged from slight (K = 0.15, 95% confidence interval [CI] = −0.19, 0.48) to poor (K = −0.06, 95% CI = −0.22, 0.10). Reliability for the overall score was poor (K = −0.004, 95% CI = −0.11, 0.11).ConclusionsDifferences in assessment and low agreement between reviewers and authors suggest the need to contact authors for information not published in studies when applying the NOS in systematic reviews.

[1]  Arie Ben-David,et al.  Comparison of classification accuracy using Cohen's Weighted Kappa , 2008, Expert Syst. Appl..

[2]  Lisa Hartling,et al.  Testing the Newcastle Ottawa Scale showed low reliability between individual reviewers. , 2013, Journal of clinical epidemiology.

[3]  Gordon H Guyatt,et al.  An observational study found that authors of randomized controlled trials frequently use concealment of randomization and blinding, despite the failure to report these methods. , 2004, Journal of clinical epidemiology.

[4]  P. Tugwell,et al.  The Newcastle-Ottawa Scale (NOS) for Assessing the Quality of Nonrandomised Studies in Meta-Analyses , 2014 .

[5]  R. Günther,et al.  Sixteen-slice spiral CT versus MR imaging for the assessment of left ventricular function in acute myocardial infarction , 2005, European Radiology.

[6]  B. Djulbegovic,et al.  Bad reporting does not mean bad methods for randomised trials: observational study of randomised controlled trials performed by the Radiation Therapy Oncology Group , 2004, BMJ : British Medical Journal.

[7]  M. Oremus,et al.  Inter-rater and test–retest reliability of quality assessments by novice student raters using the Jadad and Newcastle–Ottawa Scales , 2012, BMJ Open.

[8]  J. Higgins,et al.  Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0. The Cochrane Collaboration , 2013 .

[9]  J. Higgins.  Cochrane handbook for systematic reviews of interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration , 2011 .

[10]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[11]  Mark Loeb,et al.  Populations at risk for severe or complicated influenza illness: systematic review and meta-analysis , 2013, BMJ.

[12]  Jan-Willem Strijbos,et al.  Content analysis: What are they talking about? , 2006, Comput. Educ..