Bayesian Statistical Model of Item Response Theory in Observer Studies of Radiologists.

RATIONALE AND OBJECTIVES The purpose of this study was to validate a Bayesian statistical model of item response theory (IRT). IRT was used to evaluate a new modality (temporal subtraction, TS) in observer studies of radiologists, compared with a conventional modality (computed tomography). MATERIALS AND METHODS From previously published papers, we obtained two datasets of clinical observer studies of radiologists. Those studies used a multi-reader and multi-case paradigm to evaluate radiologists' detection abilities, primarily to determine if TS could enhance the detectability of bone metastasis or brain infarctions. We applied IRT to these studies' datasets using Stan software. Before applying IRT, the radiologists' responses were recorded as binaries for each case (1 = correct, 0 = incorrect). Effect of TS on detectability was evaluated by using our IRT model and calculating the 95% credible interval of the effect. RESULTS The mean, median, and 95% credible interval of the effect of TS were 0.913, 0.885, and 0.243-1.745 for the bone metastasis detection, and 2.524, 2.50, and 1.827-3.310, for the brain infarction detection. For both detection studies, the 95% credible intervals of the effect of TS did not include zero, indicating that TS significantly improved diagnostic ability. CONCLUSION Judgments based on the present study results were compatible with the two previous studies. Our study results demonstrated that the Bayesian statistical model of IRT could judge a new modality's usefulness.

[1]  Jiqiang Guo,et al.  Stan: A Probabilistic Programming Language. , 2017, Journal of statistical software.

[2]  Susumu Mori,et al.  Temporal Subtraction of Serial CT Images with Large Deformation Diffeomorphic Metric Mapping in the Identification of Bone Metastases. , 2017, Radiology.

[3]  Shinji Yamamoto,et al.  A method of ROC analysis by applying item response theory (IRT) to results of 1/0 judgments on the presence or absence of abnormal findings in CT image readings , 2008, SPIE Medical Imaging.

[4]  Andrew Thomas,et al.  WinBUGS - A Bayesian modelling framework: Concepts, structure, and extensibility , 2000, Stat. Comput..

[5]  RON D. HAYS,et al.  Item Response Theory and Health Outcomes Measurement in the 21st Century , 2000, Medical care.

[6]  J R Fielding,et al.  Bayesian regression methodology for estimating a receiver operating characteristic curve with two radiologic applications: prostate biopsy and spiral CT of ureteral stones. , 2001, Academic radiology.

[7]  Takeshi Kubo,et al.  Detection of suspected brain infarctions on CT can be significantly improved with temporal subtraction images , 2018, European Radiology.

[8]  A James O'Malley,et al.  Bayesian multivariate hierarchical transformation models for ROC analysis , 2006, Statistics in medicine.

[9]  Sarah Depaoli,et al.  Just Another Gibbs Sampler (JAGS) , 2016 .

[10]  Ying Lu,et al.  Indeterminate ovarian mass at US: incremental value of second imaging test for characterization--meta-analysis and Bayesian analysis. , 2005, Radiology.

[11]  Yong Luo,et al.  Using the Stan Program for Bayesian Item Response Theory , 2018, Educational and psychological measurement.

[12]  Jean-Paul Fox,et al.  Multilevel IRT Modeling in Practice with the Package mlirt , 2007 .

[13]  N. Lazar,et al.  The ASA Statement on p-Values: Context, Process, and Purpose , 2016 .

[14]  H M Bonél,et al.  Prediction of deep myometrial invasion in patients with endometrial cancer: clinical utility of contrast-enhanced MR imaging-a meta-analysis and Bayesian analysis. , 2000, Radiology.