Data from the Osteoarthritis Initiative Scoring System for Bone Marrow Lesions According to the OMERACT Filter : Validation of a Knowledge Transfer Tool for the Knee Inflammation MRI

Objective. To assess feasibility and reliability of scoring bone marrow lesions (BML) on knee magnetic resonance imaging (MRI) in osteoarthritis using the Outcome Measures in Rheumatology Knee Inflammation MRI Scoring System (KIMRISS), with a Web-based interface and online training with real-time iterative calibration. Methods. Six readers new to the KIMRISS (3 radiologists, 3 rheumatologists) scored sagittal T2-weighted fat-saturated MRI in 20 subjects randomly selected from the Osteoarthritis Initiative data, at baseline and 1-year followup. In the KIMRISS, the reader moves a transparent overlay grid within a Web-based interface to fit bones, then clicks or touches each region containing BML per slice, to score 1 if BML is present. Regional and total scores are automatically calculated. Outcomes include the interreader intraclass correlation coefficients (ICC) and the smallest detectable change (SDC). Results. Scoring took 3–12 min per scan and all readers rated the process as moderately to very user friendly. Despite a low BML burden (average score 2.8% of maximum possible) and small changes, interobserver reliability was moderate to high for BML status and change in the femur and tibia (ICC 0.78–0.88). Four readers also scored the patella reliably, whereas 2 readers were outliers, likely because of image artifacts. SDC of 1.5–5.6 represented 0.7% of the maximum possible score. Conclusion.We confirmed feasibility of knee BML scoring by new readers using interactive training and a Web-based touch-sensitive overlay system, finding high reliability and sensitivity to change. Further work will include adjustments to training materials regarding patellar scoring, and study in therapeutic trial datasets with higher burden of BML and larger changes. (J Rheumatol First Release April 1 2017; doi:10.3899/jrheum.161102) Key Indexing Terms: KNEE JOINT OSTEOARTHRITIS MRI SCORING METHODS OMERACT BONE MARROW LESION From the Department of Radiology and Diagnostic Imaging, and the Division of Rheumatology, Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada; Division of Medicine, University of New South Wales, Sydney, Australia; Department of Rheumatology, Diakonhjemmet Hospital, Oslo, Norway; Department of Radiology and Medical Imaging, Ghent University Hospital, Ghent, Belgium; King Christian 10th Hospital for Rheumatic Diseases, Gråsten; Institute of Regional Health Research, University of Southern Denmark, Odense, Denmark; Department of Radiology, Robert Jones and Agnes Hunt Orthopaedic Hospital, Oswestry, UK; Department of Radiology, Balgrist University Hospital, Zurich, Switzerland. Supported by the Capital Health Chair in Diagnostic Imaging. Dr. Jaremko is the Capital Health Chair in Diagnostic Imaging at the University of Alberta. J.L. Jaremko, MD, PhD, FRCPC, Department of Radiology and Diagnostic Imaging, Faculty of Medicine and Dentistry, University of Alberta; O. Azmat, MB, FRCP, Department of Radiology and Diagnostic Imaging, Faculty of Medicine and Dentistry, University of Alberta; R.G. Lambert, MB, FRCPC, Department of Radiology and Diagnostic Imaging, Faculty of Medicine and Dentistry, University of Alberta; P. Bird, MD, Division of Medicine, University of New South Wales; I.K. Haugen, MD, PhD, Department of Rheumatology, Diakonhjemmet Hospital; L. Jans, MD, PhD, Department of Radiology and Medical Imaging, Ghent University Hospital; U. Weber, MD, King Christian 10th Hospital for Rheumatic Diseases, and Institute of Regional Health Research, University of Southern Denmark; N. Winn, MBBS, FRCR, Department of Radiology, Robert Jones and Agnes Hunt Orthopaedic Hospital; V. Zubler, MD, Department of Radiology, Balgrist University Hospital; W.P. Maksymowych, MB ChB, FRCP(C), FACP, Division of Rheumatology, Faculty of Medicine and Dentistry, University of Alberta. Address correspondence to Dr. J.L. Jaremko, Radiologist and Assistant Professor, Department of Radiology and Diagnostic Imaging, Faculty of Medicine, University of Alberta, 2A2.41 WMC, 8440 – 112 St. NW, Edmonton, Alberta T6G 2B7, Canada. E-mail: jjaremko@ualberta.ca Accepted for publication February 14, 2017. With emerging knee osteoarthritis (OA) therapies, it is increasingly important to objectively quantify disease status and treatment response. Magnetic resonance imaging (MRI)– based semiquantitative scoring systems generally assess knee OA by whole-organ approach [Boston Leeds Osteoarthritis Knee Score (BLOKS)1, MRI Osteoarthritis Knee Score (MOAKS)2, Whole Organ magnetic Resonance iMaging Score (WORMS)3]. Scoring can be time-consuming, and Rheumatology The Journal of on April 10, 2017 Published by www.jrheum.org Downloaded from features of structural damage may change little during clinical trials, where prognosis may relate more to active disease involving bone marrow lesions (BML)4. We developed the Outcome Measures in Rheumatology (OMERACT) Knee Inflammation MRI Scoring System (KIMRISS) using a wide scoring range to optimize sensitivity to change, binary scoring to improve reliability, and an integrated standardized new-reader calibration method to optimize feasibility. BML is typically scored separately for multiple peri articular bone regions. KIMRISS scoring is binary: Is BML present? Yes/No (1/0). The BLOKS/WORMS/MOAKS1,2,3 involve more complex decisions, estimating percentages of fewer large regions containing BML. New readers train by reading published manuscripts and performing in-person exercises with experienced readers. In contrast, we developed a novel approach allowing systematic reader calibration without direct expert supervision. In this real-time iterative calibration (RETIC), readers use a Web-based digital overlay superimposing the outline of scoring regions on MRI. As the reader completes the scoring of each case, each overlay region changes color, indicating concordance or discordance with expert scores. This interactive feedback allows new users to rapidly align their scoring with experts as they progress through cases. We tested the feasibility and reliability of the KIMRISS knee OA BML scoring using Web-based digital overlay and RETIC technology. We applied the relevant aspects of the OMERACT Filter 2.05,6. Members of a subgroup of the OMERACT MRI in the Arthritis Working Group performed a reading exercise from March–April 2016 and presented the results at OMERACT 13 (Whistler, British Columbia, Canada, May 2016). In accordance with the OMERACT handbook7, no previous calibration tools were found in a literature review by a fellow in this group, which in agreement with the OMERACT executive committee included clinical professionals, methodologists, and health care professionals. MATERIALS AND METHODS Interactive touch-sensitive, Web-based interface. In OMERACT 12, we demonstrated that a digital image overlay for hip OA BML scoring (by the Hip Inflammation MRI Scoring System) had reliability equivalent to conventional methods and was preferred by readers8,9. We developed a Web-based interface suitable for use on touch-sensitive screens, and generated a knee-joint–specific overlay for femur, tibia, and patella. Readers upload or open a sagittal fluid-sensitive [intermediate-weighted fat-saturated (IWFS) or short-tau inversion recovery] knee MRI sequence online (www.carearthritis.com, “Osteoarthritis Imaging,” accounts free to registered users). The reader resizes and moves a transparent grid overlay to fit each bone on the central slice where anterior/posterior cruciate ligaments cross, then scrolls through all slices, touching or mouse-clicking each overlay region containing BML. This causes shading to appear (Figure 1), and the Web tool records a score of 1 to indicate that it has been selected. Upon scoring completion, the Web tool outputs a spreadsheet file containing scores 0/1 per region and summary statistics. The maximum possible KIMRISS score across 29 slices includes 763 regions (290 tibia, 377 femur, 96 patella), although no actual knee has BML this extensive. Because regions of templates falling outside bone were not scored, total possible scores will vary slightly for individual patients depending on bone size. RETIC tool. We provided readers with an instructional slide presentation including a scoring atlas giving examples of true BML versus confounders such as volume averaging at condylar edges, subchondral cystic change, and hemato poietic marrow; a video demonstrating KIMRISS scoring (youtu.be/k988FmLVhb0); and the new Web-based RETIC tool. In RETIC training mode, the reader scores cases previously scored by experts. When the reader has finished selecting positive regions, the overlay changes color in each region indicating whether reader and expert scores are concordant/discordant. Intraclass correlation coefficients (ICC) between reader and experts are instantly displayed, allowing real-time calibration and rapid progressive learning with each new case. For RETIC training, 8 cases (2 timepoints each) from the Osteoarthritis Initiative (OAI) data were scored by 2 experienced KIMRISS developers, with discrepancies resolved by consensus. Data. We used publicly available data from the OAI (www.oai.ucsf.edu, v.18). This was a multicenter, prospective observational study of knee OA10, in which 4796 men and women aged 45–79 years enrolled between 2004 and 2006 underwent annual knee assessment including MRI. We randomly selected 1 knee for each of 20 randomly selected subjects with imaging and clinical information available at baseline and 1-year followup. We scored sagittal IWFS images (repetition/echo times TR/TE 3200/30 ms, matrix 444 × 448, slice thickness 3 mm, field-of-view 159 × 160 mm). Reading. We had 6 readers: 3 musculoskeletal radiologists and 3 rheumatologists (7–30+ yrs of experience). Only 1 reader had previously scored the KIMRISS. Each reader scored BML at tibia, femur, and patella for each selected knee at both timepoints once. Scans from the 2 timepoints were pre