Development of a diagnostic test set to assess agreement in breast pathology: practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS)

BackgroundDiagnostic test sets are a valuable research tool that contributes importantly to the validity and reliability of studies that assess agreement in breast pathology. In order to fully understand the strengths and weaknesses of any agreement and reliability study, however, the methods should be fully reported. In this paper we provide a step-by-step description of the methods used to create four complex test sets for a study of diagnostic agreement among pathologists interpreting breast biopsy specimens. We use the newly developed Guidelines for Reporting Reliability and Agreement Studies (GRRAS) as a basis to report these methods.MethodsBreast tissue biopsies were selected from the National Cancer Institute-funded Breast Cancer Surveillance Consortium sites. We used a random sampling stratified according to woman’s age (40–49 vs. ≥50), parenchymal breast density (low vs. high) and interpretation of the original pathologist. A 3-member panel of expert breast pathologists first independently interpreted each case using five primary diagnostic categories (non-proliferative changes, proliferative changes without atypia, atypical ductal hyperplasia, ductal carcinoma in situ, and invasive carcinoma). When the experts did not unanimously agree on a case diagnosis a modified Delphi method was used to determine the reference standard consensus diagnosis. The final test cases were stratified and randomly assigned into one of four unique test sets.ConclusionsWe found GRRAS recommendations to be very useful in reporting diagnostic test set development and recommend inclusion of two additional criteria: 1) characterizing the study population and 2) describing the methods for reference diagnosis, when applicable.

[1]  S J Schnitt,et al.  Interobserver Reproducibility in the Diagnosis of Ductal Proliferative Breast Lesions Using Standardized Criteria , 1992, The American journal of surgical pathology.

[2]  J. Sztajzel,et al.  Accuracy of cardiac auscultation in the era of Doppler-echocardiography: a comparison between cardiologists and internists. , 2010, International journal of cardiology.

[3]  A. Hrõbjartsson,et al.  Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. , 2011, Journal of clinical epidemiology.

[4]  Herbert Y Kressel,et al.  Consensus interpretation in imaging research: is there a better way? , 2010, Radiology.

[5]  Luisa P. Wallace,et al.  The "laboratory" effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations. , 2008, Radiology.

[6]  S C Weller,et al.  Assessing Rater Performance without a "Gold Standard" Using Consensus Theory , 1997, Medical decision making : an international journal of the Society for Medical Decision Making.

[7]  W. Barlow,et al.  Current medicolegal and confidentiality issues in large, multicenter research programs. , 2000, American journal of epidemiology.

[8]  David Moher,et al.  The STARD Statement for Reporting Studies of Diagnostic Accuracy: Explanation and Elaboration , 2003, Annals of Internal Medicine [serial online].

[9]  A R Feinstein,et al.  Context bias. A problem in diagnostic radiology. , 1996, JAMA.

[10]  J. Elmore,et al.  Variability in radiologists' interpretations of mammograms. , 1994, The New England journal of medicine.

[11]  Qingsong Xu,et al.  Columnar cell lesions: a consensus study among pathology trainees. , 2010, Human pathology.

[12]  M. Linver 4–23 Pathologic Findings From the Breast Cancer Surveillance Consortium: Population-Based Outcomes in Women Undergoing Biopsy After Screening Mammography , 2007 .

[13]  Hiroshi Yano,et al.  Diagnostic performance of ADC for Non-mass-like breast lesions on MR imaging. , 2010, Magnetic resonance in medical sciences : MRMS : an official journal of Japan Society of Magnetic Resonance in Medicine.

[14]  K. Kerlikowske,et al.  Breast Cancer Surveillance Consortium: a national mammography screening and outcomes database. , 1997, AJR. American journal of roentgenology.

[15]  Baljit Singh,et al.  Atypia on breast core needle biopsies: reproducibility and significance. , 2009, Annals of clinical and laboratory science.

[16]  N F Boyd,et al.  Mammographic density, lobular involution, and risk of breast cancer , 2008, British Journal of Cancer.

[17]  H. Honest,et al.  Reporting of measures of accuracy in systematic reviews of diagnostic literature , 2002, BMC health services research.

[18]  Johannes B Reitsma,et al.  Quality of reporting of diagnostic accuracy studies. , 2005, Radiology.

[19]  P. Summers,et al.  Intraobserver and interobserver variability in the calculation of apparent diffusion coefficient (ADC) from diffusion-weighted magnetic resonance imaging (DW-MRI) of breast tumours , 2011, La radiologia medica.

[20]  V. McCormack,et al.  Issues in the reporting of epidemiological studies: a survey of recent practice , 2004, BMJ : British Medical Journal.

[21]  Karla Kerlikowske,et al.  Pathologic findings from the Breast Cancer Surveillance Consortium , 2006, Cancer.

[22]  Robert A. Goulart,et al.  Diagnostic Agreement in the Evaluation of Image-guided Breast Core Needle Biopsies: Results from a Randomized Clinical Trial , 2004, The American journal of surgical pathology.

[23]  H. Kundel,et al.  Measurement of observer agreement. , 2003, Radiology.

[24]  B Littenberg,et al.  The New Hampshire Mammography Network: the development and design of a population-based registry. , 1996, AJR. American journal of roentgenology.

[25]  P. Bossuyt,et al.  The quality of diagnostic accuracy studies since the STARD statement , 2006, Neurology.

[26]  J. Skelly,et al.  Predicting Biopsy Outcome After Mammography: What Is the Likelihood the Patient Has Invasive or In Situ Breast Cancer? , 2005, Annals of Surgical Oncology.

[27]  A. Tosteson,et al.  Statewide study of diagnostic agreement in breast pathology. , 1998, Journal of the National Cancer Institute.

[28]  J. Rosai,et al.  Borderline Epithelial Lesions of the Breast , 1991, The American journal of surgical pathology.