A unified framework for bias assessment in clinical research.

Methodological flaws, limitations, and inadequate practices in research are well known and pose threats to the internal validity of any research study. However, there are ways of safeguarding research conduct to reduce the chance of research producing distorted results. Numerous tools now exist to assess the incorporation of such safeguards into primary research studies (also known as quality and/or risk-of-bias assessment). These tools typically include a variety of items that are then checked against those implemented in the study. Despite a lot of research in this area, no comprehensive generic classification of safeguards across study designs exist, although attempts have been made to clarify aspects of this. We review the developments in this area as well as use preliminary data from 100 methodological studies to illustrate our proposed approach. We conclude by proposing a new framework for identifying research studies at risk of being biased and the information in this article will promote a unification of the diverse approaches to facilitating bias assessment in clinical research.

[1]  J. A. Smith,et al.  The problem of appraising qualitative research , 2004, Quality and Safety in Health Care.

[2]  A. Michalsen,et al.  Checklist for the qualitative evaluation of clinical studies with particular focus on external validity and model validity , 2006, BMC medical research methodology.

[3]  A. Cipriani,et al.  Relationship Between Quality of Reports of Antidepressant Randomized Controlled Trials and Treatment Estimates: Systematic Review, Meta-Analysis, and Meta-Regression Analysis , 2007, Journal of clinical psychopharmacology.

[4]  A. Raymer,et al.  Systematic review of communication partner training in aphasia: Methodological quality , 2013, International journal of speech-language pathology.

[5]  A. Genaidy,et al.  An epidemiological appraisal instrument – a tool for evaluation of epidemiological studies , 2007, Ergonomics.

[6]  S. Harris,et al.  Physical therapist management of lymphedema following treatment for breast cancer: a critical review of its effectiveness. , 1998, Physical therapy.

[7]  P. Santaguida,et al.  Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions , 2012 .

[8]  M. Delgado-Rodríguez,et al.  Bias , 2004, Journal of Epidemiology and Community Health.

[9]  H. Saltaji,et al.  Poor Reliability between Cochrane Reviewers and Blinded External Reviewers When Applying the Cochrane Risk of Bias Tool in Physical Therapy Trials , 2014, PLoS ONE.

[10]  G. Muschler,et al.  Intra-Articular Cellular Therapy for Osteoarthritis and Focal Cartilage Defects of the Knee: A Systematic Review of the Literature and Study Quality Analysis. , 2016, The Journal of bone and joint surgery. American volume.

[11]  Mary Godfrey,et al.  An evaluation tool to assess the quality of qualitative research studies , 2004 .

[12]  M. Feinleib,et al.  A positive or a negative confounding variable? A simple teaching aid for clinicians and students. , 2005, Annals of epidemiology.

[13]  T C Chalmers,et al.  Bias in treatment assignment in controlled clinical trials. , 1983, The New England journal of medicine.

[14]  N. Higginbotham,et al.  Critical appraisal checklist for qualitative research studies , 2000, Indian journal of pediatrics.

[15]  R. Moore,et al.  Teasing apart quality and validity in systematic reviews: an example from acupuncture trials in chronic neck and back pain , 2000, Pain.

[16]  P. Fisher,et al.  A systematic review of the quality of homeopathic pathogenetic trials published from 1945 to 1995 , 2007, Homeopathy.

[17]  C. Moga,et al.  A principal component analysis is conducted for a case series quality appraisal checklist. , 2016, Journal of clinical epidemiology.

[18]  J. Kleijnen,et al.  Clinical trials of homoeopathy. , 1991, BMJ.

[19]  Christopher Eccleston,et al.  A scale for rating the quality of psychological trials for pain , 2005, Pain.

[20]  A. Feinstein XLVIII. Efficacy of different research structures in preventing bias in the analysis of causation , 1979, Clinical pharmacology and therapeutics.

[21]  D. Ring,et al.  Quality of prospective controlled randomized trials. Analysis of trials of treatment for lateral epicondylitis as an example. , 2007, The Journal of bone and joint surgery. American volume.

[22]  P. Tugwell,et al.  Stratification by quality induced selection bias in a meta-analysis of clinical trials. , 2019, Journal of clinical epidemiology.

[23]  C L Beaudin,et al.  Critical appraisal tools facilitate the work of the quality professional. , 2001, Journal for healthcare quality : official publication of the National Association for Healthcare Quality.

[24]  J. Robins,et al.  A Structural Approach to Selection Bias , 2004, Epidemiology.

[25]  L. Gluud Bias in clinical intervention research. , 2006, American journal of epidemiology.

[26]  C D Naylor,et al.  Incorporating variations in the quality of individual randomized trials into meta-analysis. , 1992, Journal of clinical epidemiology.

[27]  D. Sackett,et al.  An empirical demonstration of Berkson's bias. , 1978, Journal of chronic diseases.

[28]  David Moher,et al.  A checklist to evaluate a report of a nonpharmacological trial (CLEAR NPT) was developed using consensus. , 2005, Journal of clinical epidemiology.

[29]  F. Song,et al.  Evaluating non-randomised intervention studies. , 2003, Health technology assessment.

[30]  D. Thijssen,et al.  Assessing the perceived quality of brachial artery Flow Mediated Dilation studies for inclusion in meta-analyses and systematic reviews: Description of data employed in the development of a scoring ;tool based on currently accepted guidelines , 2016, Data in brief.

[31]  Alex H. S. Harris,et al.  ACL Reconstruction in Patients Aged 40 Years and Older , 2013, The American journal of sports medicine.

[32]  S D Walter,et al.  An analysis of Berkson's bias in case-control studies. , 1986, Journal of chronic diseases.

[33]  Joyce A. Mitchell,et al.  Methods of Randomized Controlled Clinical Trials in Health Services Research , 1995, Medical care.

[34]  K. Raphael,et al.  Development of a quality-assessment tool for experimental bruxism studies: reliability and validity. , 2013, Journal of orofacial pain.

[35]  Lindsay Glynn,et al.  A critical appraisal tool for library and information research , 2006, Libr. Hi Tech.

[36]  Andrew J. Vickers,et al.  Critical appraisal: How to read a clinical research paper , 1995 .

[37]  H. Polo Friz,et al.  Thromboprophylaxis with Low-Molecular-Weight Heparins: An Assessment of the Methodological Quality of Studies , 2015, Seminars in Thrombosis & Hemostasis.

[38]  F. Meijman,et al.  The extent of inter- and intrareviewer agreement on the classification and assessment of designs of single-practice research. , 1995, Family practice.

[39]  S. Doi Understanding Evidence in Health Care: Using Clinical Epidemiology , 2011 .

[40]  T C Chalmers,et al.  Randomized versus historical controls for clinical trials. , 1982, The American journal of medicine.

[41]  P. Katrak,et al.  A systematic review of the content of critical appraisal tools , 2004, BMC medical research methodology.

[42]  Leanne Togher,et al.  Rating the methodological quality of single-subject designs and n-of-1 trials: Introducing the Single-Case Experimental Design (SCED) Scale , 2008, Neuropsychological rehabilitation.

[43]  T C Chalmers,et al.  A method for assessing the quality of a randomized control trial. , 1981, Controlled clinical trials.

[44]  F. Blyth,et al.  Assessing risk of bias in prevalence studies: modification of an existing tool and evidence of interrater agreement. , 2012, Journal of clinical epidemiology.

[45]  Denis Walsh,et al.  Appraising the quality of qualitative research. , 2006, Midwifery.

[46]  D. Rennie,et al.  Measuring the quality of trials: the quality of quality scales. , 1999, JAMA.

[47]  M. Egger,et al.  The hazards of scoring the quality of clinical trials for meta-analysis. , 1999, JAMA.

[48]  E Andrew,et al.  Method for Assessment of the Reporting Standard of Clinical Trials with Roentgen Contrast Media , 1984, Acta radiologica: diagnosis.

[49]  L M Bouter,et al.  Spinal manipulation and mobilisation for back and neck pain: a blinded review. , 1991, BMJ.

[50]  N. Dreyer,et al.  The GRACE Checklist: A Validated Assessment Tool for High Quality Observational Studies of Comparative Effectiveness , 2016, Journal of managed care & specialty pharmacy.

[51]  S. Sheen,et al.  Testing a tool for assessing the risk of bias for nonrandomized studies showed moderate reliability and promising validity. , 2013, Journal of clinical epidemiology.

[52]  H. Saltaji,et al.  PEDro or Cochrane to Assess the Quality of Clinical Trials? A Meta-Epidemiological Study , 2015, PloS one.

[53]  H. Saltaji,et al.  Identifying Items to Assess Methodological Quality in Physical Therapy Trials: A Factor Analysis , 2014, Physical Therapy.

[54]  B. Koes,et al.  Exercise therapy for low back pain. , 2000, The Cochrane database of systematic reviews.

[55]  Eva A Rehfuess,et al.  Quality appraisal in systematic reviews of public health interventions: an empirical study on the impact of choice of tool on meta-analysis , 2012, Journal of Epidemiology & Community Health.

[56]  P. Shekelle,et al.  Evidence assessment of management of acute otitis media: II. Research gaps and priorities for future research. , 2001, Pediatrics.

[57]  B. Prahl-Andersen,et al.  A systematic review of the relationship between overjet size and traumatic dental injuries. , 1999, European journal of orthodontics.

[58]  Lorelei Lingard,et al.  Critically appraising qualitative research , 2008, BMJ : British Medical Journal.

[59]  D. Cook,et al.  Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? , 1998, The Lancet.

[60]  P. Gøtzsche Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis. , 1989, Controlled clinical trials.

[61]  Peter C Gøtzsche,et al.  Bmc Medical Research Methodology Open Access Recommendations by Cochrane Review Groups for Assessment of the Risk of Bias in Studies , 2022 .

[62]  R. J. Hayes,et al.  Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. , 1995, JAMA.

[63]  E. Power,et al.  Strengthening the quality of longitudinal research into cognitive-communication recovery after traumatic brain injury: A systematic review , 2017, International journal of speech-language pathology.

[64]  L W Chambers,et al.  Critical appraisal of the health research literature: prevalence or incidence of a health problem. , 1998, Chronic diseases in Canada.

[65]  L Arrivé,et al.  A scale of methodological quality for clinical studies of radiologic examinations. , 2000, Radiology.

[66]  Isabelle Boutron,et al.  A revised tool for assessing risk of bias in randomized trials , 2016 .

[67]  K. Seers,et al.  Development of a tool to rate the quality assessment of randomized controlled trials using a Delphi technique. , 1997, Journal of advanced nursing.

[68]  M. Duffy,et al.  A research appraisal checklist for evaluating nursing research reports. , 1985, Nursing & health care : official publication of the National League for Nursing.

[69]  W. Jonas,et al.  A systematic review of the quality of research on hands-on and distance healing: clinical and laboratory studies. , 2003, Alternative Therapies in Health and Medicine.

[70]  Rachel Churchill,et al.  Development of a quality assessment instrument for trials of treatments for depression and neurosis , 2001 .

[71]  L Abenhaim,et al.  Links between passive smoking and disease: a best-evidence synthesis. A report of the Working Group on Passive Smoking. , 1990, Clinical and investigative medicine. Medecine clinique et experimentale.

[72]  H Checkoway,et al.  Bias due to misclassification in the estimation of relative risk. , 1977, American journal of epidemiology.

[73]  P. Vineis History of bias , 2005, Sozial- und Präventivmedizin.

[74]  A R Jadad,et al.  Assessing the quality of reports of randomized clinical trials: is blinding necessary? , 1996, Controlled clinical trials.

[75]  A. Keller,et al.  Sugar-Sweetened Beverages and Obesity Risk in Children and Adolescents: A Systematic Analysis on How Methodological Quality May Influence Conclusions. , 2016, Journal of the Academy of Nutrition and Dietetics.

[76]  R. Teasell,et al.  Quality of the Stroke Rehabilitation Research , 2003, Topics in stroke rehabilitation.

[77]  Tasha R. Stanton,et al.  Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review , 2008, Physical Therapy.

[78]  J. Tyson,et al.  Aid to the evaluation of therapeutic studies. , 1989, Pediatrics.

[79]  S. Chacón-Moscoso,et al.  The Development of a Checklist to Enhance Methodological Quality in Intervention Programs , 2016, Front. Psychol..

[80]  M. Evans,et al.  A score system for evaluating random control clinical trials of prophylaxis of abdominal surgical wound infection , 1985, The British journal of surgery.

[81]  G H Guyatt,et al.  Respiratory muscle training in chronic airflow limitation: a meta-analysis. , 1992, The American review of respiratory disease.

[82]  K. Schulz,et al.  Bias and causal associations in observational research , 2002, The Lancet.

[83]  M. Thase,et al.  A new scale for assessing the quality of randomized clinical trials of psychotherapy. , 2010, Comprehensive psychiatry.

[84]  Daniel Westreich,et al.  Berkson's bias, selection bias, and missing data. , 2012, Epidemiology.

[85]  R H DuRant,et al.  Checklist for the evaluation of research articles. , 1994, The Journal of adolescent health : official publication of the Society for Adolescent Medicine.

[86]  N. Black,et al.  The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. , 1998, Journal of epidemiology and community health.

[87]  H. Vet,et al.  The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. , 1998, Journal of clinical epidemiology.

[88]  P Peduzzi,et al.  Analysis as-randomized and the problem of non-adherence: an example from the Veterans Affairs Randomized Trial of Coronary Artery Bypass Surgery. , 1993, Statistics in medicine.

[89]  J. Urschel,et al.  How To Analyze an Article , 2005, World Journal of Surgery.

[90]  Arpana Verma,et al.  Critical appraisal for public health: a new checklist. , 2008, Public health.

[91]  Xian-tao Zeng,et al.  The methodological quality assessment tools for preclinical and clinical studies, systematic review and meta‐analysis, and clinical practice guideline: a systematic review , 2015, Journal of evidence-based medicine.

[92]  Bernadette Mazurek Melnyk,et al.  Rapid critical appraisal of randomized controlled trials (RCTs): an essential skill for evidence-based practice (EBP). , 2005, Pediatric nursing.

[93]  Patrick Onghena,et al.  Antidepressant-induced analgesia in chronic non-malignant pain: a meta-analysis of 39 placebo-controlled studies , 1992, Pain.

[94]  J. Higgins,et al.  Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. , 2007, International journal of epidemiology.

[95]  Luc Côté,et al.  Appraising qualitative research articles in medicine and medical education , 2005, Medical teacher.

[96]  J. Sterne,et al.  The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials , 2011, BMJ : British Medical Journal.

[97]  D. Sackett Bias in analytic research. , 1979, Journal of chronic diseases.

[98]  Iveta Simera,et al.  A proposed framework for developing quality assessment tools , 2017, Systematic Reviews.

[99]  G. Guyatt,et al.  Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. , 2010, JAMA.

[100]  A. Murray,et al.  Developing a quality scoring system for epidemiological surveys of genetic disorders , 2002, Clinical genetics.

[101]  David Kriebel,et al.  Bias in occupational epidemiology studies , 2006, Occupational and Environmental Medicine.

[102]  S Greenland,et al.  On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. , 2001, Biostatistics.

[103]  Paul Knipschild,et al.  Systematic Reviews on the Basis of Methodological Criteria , 1997 .

[104]  Andrea Furlan,et al.  Updated Method Guidelines for Systematic Reviews in the Cochrane Collaboration Back Review Group , 2003, Spine.

[105]  Ethan M Balk,et al.  Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials. , 2002, JAMA.

[106]  K. Morin,et al.  Evaluating the level of evidence of qualitative research. , 2002, Journal of obstetric, gynecologic, and neonatal nursing : JOGNN.

[107]  B. H. Thomas,et al.  A process for systematically reviewing the literature: providing the research evidence for public health nursing interventions. , 2004, Worldviews on evidence-based nursing.

[108]  Harris Cooper,et al.  A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: the Study Design and Implementation Assessment Device (Study DIAD). , 2008, Psychological methods.

[109]  M. Cho,et al.  Instruments for assessing the quality of drug studies published in the medical literature. , 1994, JAMA.

[110]  John P A Ioannidis,et al.  Commentary: Adjusting for bias: a user's guide to performing plastic surgery on meta-analyses of observational studies. , 2011, International journal of epidemiology.

[111]  L. Mainwaring,et al.  Evaluating the methodological quality of sports neuropsychology concussion research: A systematic review , 2010, Brain injury.

[112]  L. Bouter,et al.  Scoring the quality of clinical trials. , 2000, JAMA.

[113]  D. Ring,et al.  Quality of prospective controlled randomized trials. Analysis of trials of treatment for lateral epicondylitis as an example. , 2007, The Journal of bone and joint surgery. American volume.

[114]  Ian Shrier,et al.  Structural Approach to Bias in Meta‐analyses , 2011, Research synthesis methods.

[115]  J. Vandenbroucke,et al.  Low-molecular-weight heparin versus standard heparin in general and orthopaedic surgery: a meta-analysis , 1992, The Lancet.

[116]  M. Crowe,et al.  Comparison of the effects of using the Crowe Critical Appraisal Tool versus informal appraisal in assessing health research: a randomised trial. , 2011, International journal of evidence-based healthcare.

[117]  Maurilio Marcacci,et al.  Matrix-Assisted Autologous Chondrocyte Transplantation for the Repair of Cartilage Defects of the Knee , 2009, The American journal of sports medicine.

[118]  S. Greenland Quality Scores Are Useless and Potentially Misleading: Reply to “Re: A Critical Look at Some Popular Analytic Methods” , 1994 .

[119]  Gail M. Williams,et al.  Advances in the meta-analysis of heterogeneous clinical trials II: The quality effects model. , 2015, Contemporary clinical trials.

[120]  Penny Whiting,et al.  Bmc Medical Research Methodology Open Access No Role for Quality Scores in Systematic Reviews of Diagnostic Accuracy Studies , 2005 .

[121]  Suhail A R Doi,et al.  A Quality-Effects Model for Meta-Analysis , 2008, Epidemiology.

[122]  JoDee Anderson,et al.  Development of a quality assessment scale for retrospective clinical studies in pediatric surgery. , 2003, Journal of pediatric surgery.

[123]  W. Zingg,et al.  Innovative tools for quality assessment: integrated quality criteria for review of multiple study designs (ICROMS). , 2016, Public health.

[124]  J. Kleijnen,et al.  Acupuncture and chronic pain: a criteria-based meta-analysis. , 1990, Journal of clinical epidemiology.

[125]  M. Schmitter,et al.  A new instrument for assessing the quality of studies on prevalence , 2011, Clinical Oral Investigations.

[126]  M. Koehoorn,et al.  Applying epidemiological principles to ergonomics: a checklist for incorporating sound design and interpretation of studies. , 1997, Applied ergonomics.

[127]  M. Hernán,et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions , 2016, British Medical Journal.

[128]  G A Colditz,et al.  How study design affects outcomes in comparisons of therapy. II: Surgical. , 1989, Statistics in medicine.

[129]  M J Lichtenstein,et al.  Guidelines for reading case-control studies. , 1987, Journal of chronic diseases.

[130]  R A Oostendorp,et al.  The efficacy of laser therapy for musculoskeletal and skin disorders: a criteria-based meta-analysis of randomized clinical trials. , 1992, Physical therapy.

[131]  L M Bouter,et al.  Method guidelines for systematic reviews in the Cochrane Collaboration Back Review Group for Spinal Disorders. , 1997, Spine.

[132]  A. Moseley,et al.  PEDro. A database of randomized trials and systematic reviews in physiotherapy. , 2000, Manual therapy.

[133]  F. Mosteller,et al.  How study design affects outcomes in comparisons of therapy. I: Medical. , 1989, Statistics in medicine.

[134]  A. Whelan,et al.  Development and evaluation of an instrument for the critical appraisal of randomized controlled trials of natural products , 2009, BMC complementary and alternative medicine.

[135]  N McKoy,et al.  Systems to rate the strength of scientific evidence. , 2002, Evidence report/technology assessment.

[136]  S. Hawker,et al.  Appraising the Evidence: Reviewing Disparate Data Systematically , 2002, Qualitative health research.

[137]  T. Imperiale,et al.  Do corticosteroids reduce mortality from alcoholic hepatitis? A meta-analysis of the randomized trials. , 1990, Annals of internal medicine.

[138]  A R Jadad,et al.  Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. , 1995, Controlled clinical trials.

[139]  J. Wark,et al.  Studies of surgical outcome after patellar tendinopathy: clinical significance of methodological deficiencies and guidelines for future studies , 2000, Scandinavian journal of medicine & science in sports.

[140]  C. Manterola,et al.  Reliability and validity of MINCIR scale for methodological quality in dental therapy research. , 2014, Brazilian oral research.

[141]  M. Egger,et al.  PEDro's bias: summary quality scores should not be used in meta-analysis. , 2013, Journal of clinical epidemiology.

[142]  M. Nielsen,et al.  A Guide to Understanding and Evaluating Research Articles , 1985 .

[143]  Douglas G Altman,et al.  Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study , 2008, BMJ : British Medical Journal.

[144]  G. Guyatt,et al.  Systematic reviewers neglect bias that results from trials stopped early for benefit. , 2007, Journal of clinical epidemiology.

[145]  P. Pluye,et al.  A scoring system for appraising mixed methods research, and concomitantly appraising qualitative, quantitative and mixed methods primary studies in Mixed Studies Reviews. , 2009, International journal of nursing studies.

[146]  Peter Herbison,et al.  Adjustment of meta-analyses on the basis of quality scores should be abandoned. , 2006, Journal of clinical epidemiology.

[147]  John D. Childs,et al.  Systematic review of the quality of randomized controlled trials for patellofemoral pain syndrome. , 2003, The Journal of orthopaedic and sports physical therapy.

[148]  Shmuel Reis,et al.  Aggregation of qualitative studies--From theory to practice: Patient priorities and family medicine/general practice evaluations. , 2007, Patient education and counseling.

[149]  Carlos Manterola,et al.  What Is the Methodologic Quality of Human Therapy Studies in ISI Surgical Publications? , 2006, Annals of surgery.