ROC curve analyses of eyewitness identification decisions: An analysis of the recent debate

How should the accuracy of eyewitness identification decisions be measured, so that best practices for identification can be determined? This fundamental question is under intense debate. One side advocates for continued use of a traditional measure of identification accuracy, known as the diagnosticity ratio, whereas the other side argues that receiver operating characteristic curves (ROCs) should be used instead because diagnosticity is confounded with response bias. Diagnosticity proponents have offered several criticisms of ROCs, which we show are either false or irrelevant to the assessment of eyewitness accuracy. We also show that, like diagnosticity, Bayesian measures of identification accuracy confound response bias with witnesses’ ability to discriminate guilty from innocent suspects. ROCs are an essential tool for distinguishing memory-based processes from decisional aspects of a response; simulations of different possible identification tasks and response strategies show that they offer important constraints on theory development.

[1]  Steven E. Clark,et al.  A memory and decision model for eyewitness identification , 2003 .

[2]  Curt A. Carlson,et al.  An Investigation of the Weapon Focus Effect and the Confidence–Accuracy Relationship for Eyewitness Identification , 2017 .

[3]  Curt A. Carlson,et al.  An evaluation of lineup presentation, weapon presence, and a distinctive feature using ROC analysis , 2014 .

[4]  John T Wixted,et al.  A signal-detection-based diagnostic-feature-detection model of eyewitness identification. , 2014, Psychological review.

[5]  David G. Dobolyi,et al.  Eyewitness confidence in simultaneous and sequential lineups: a criterion shift account for sequential mistaken identification overconfidence. , 2013, Journal of experimental psychology. Applied.

[6]  Andrew M. Smith,et al.  ROC analysis of lineups obscures information that is critical for both theoretical understanding and applied purposes , 2015 .

[7]  G. Wells,et al.  Eyewitness identification: Bayesian information gain, base-rate effect equivalency curves, and reasonable suspicion. , 2015, Law and human behavior.

[8]  M. Chun.  Scene Perception and Memory , 2003 .

[9]  R Ratcliff,et al.  Testing global memory models using ROC curves. , 1992, Psychological review.

[10]  C. Metz.  ROC Methodology in Radiologic Imaging , 1986, Investigative radiology.

[11]  Lawrence T. DeCarlo.  Signal detection models for the same–different task , 2013 .

[12]  Anders Winman,et al.  Calibration and diagnosticity of confidence in eyewitness identification: Comments on what can be inferred from the low confidence-accuracy correlation , 1996 .

[13]  C. Tredoux.  Statistical Inference on Measures of Lineup Fairness , 1998 .

[14]  C. Rotello,et al.  Does familiarity change in the revelation effect? , 2003, Journal of experimental psychology. Learning, memory, and cognition.

[15]  Laura Mickes,et al.  Receiver operating characteristic analysis and confidence–accuracy characteristic analysis in investigations of system variables and estimator variables that affect eyewitness memory , 2015 .

[16]  Caren M Rotello,et al.  “Remembering” emotional words is based on response bias, not recollection , 2007, Psychonomic bulletin & review.

[17]  K. Rayner,et al.  Scene perception and memory revealed by eye movements and ROC analysis: Does a cultural difference truly exist? , 2010 .

[18]  James Michael Lampinen,et al.  ROC analyses in eyewitness identification research , 2016 .

[19]  Neil A. Macmillan,et al.  Detection Theory: A User's Guide , 1991 .

[20]  J. Swets.  Indices of discrimination or diagnostic accuracy: their ROCs and implied models. , 1986, Psychological bulletin.

[21]  J. Wixted,et al.  The Field of Eyewitness Memory Should Abandon Probative Value and Embrace Receiver Operating Characteristic Analysis , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[22]  Caren M Rotello,et al.  Type I error rates and power analyses for single-point sensitivity measures , 2008, Perception & psychophysics.

[23]  William Wells,et al.  Estimating the reliability of eyewitness identifications from police lineups , 2015, Proceedings of the National Academy of Sciences.

[24]  J. Kantner,et al.  Cross-situational consistency in recognition memory response bias , 2014, Psychonomic bulletin & review.

[25]  Neil Brewer,et al.  The confidence-accuracy relationship for eyewitness identification decisions: Effects of exposure duration, retention interval, and divided attention. , 2013, Journal of experimental psychology. Applied.

[26]  Neil A. Macmillan,et al.  Detection theory: A user's guide, 2nd ed. , 2005 .

[27]  Steven D. Penrod,et al.  Juror decision making in eyewitness identification cases , 1988 .

[28]  Roger Ratcliff and Gail McKoon.  Using ROC Data and Priming Results to Test Global Memory Models , 2013 .

[29]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[30]  Is ROC analysis a tool that should replace probative analysis in studying lineups , 2016 .

[31]  S. Clark.  Costs and Benefits of Eyewitness Identification Reform , 2012, Perspectives on psychological science : a journal of the Association for Psychological Science.

[32]  J. Wixted,et al.  ROC analysis measures objective discriminability for any eyewitness identification procedure , 2015 .

[33]  M. Greiner,et al.  Receiver-operating characteristic curves and likelihood ratios: improvements over traditional methods for the evaluation and application of veterinary clinical pathology tests. , 2006, Veterinary clinical pathology.

[34]  John T Wixted,et al.  Receiver operating characteristic analysis of eyewitness memory: comparing the diagnostic accuracy of simultaneous versus sequential lineups. , 2012, Journal of experimental psychology. Applied.

[35]  Scott D. Gronlund,et al.  Effect of retention interval on showup and lineup performance , 2015 .

[36]  S. Gronlund,et al.  Global matching models of recognition memory: How the models match the data , 1996, Psychonomic bulletin & review.

[37]  K. Rayner,et al.  Short Article: Scene perception and memory revealed by eye movements and receiver-operating characteristic analyses: Does a cultural difference truly exist? , 2009, Quarterly journal of experimental psychology.

[38]  Neil Brewer,et al.  The effects of allowing a second sequential lineup lap on choosing and probative value , 2015 .

[39]  J. Wixted,et al.  Evaluating eyewitness identification procedures: ROC analysis and its misconceptions , 2015 .

[40]  J A Swets,et al.  Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance. , 1986, Psychological bulletin.

[41]  Andrew M. Smith,et al.  ROC analysis of lineups does not measure underlying discriminability and has limited value , 2015 .

[42]  R. Shiffrin,et al.  A model for recognition memory: REM—retrieving effectively from memory , 1997, Psychonomic bulletin & review.

[43]  Scott D. Gronlund,et al.  Showups versus lineups: An evaluation using ROC analysis , 2012 .

[44]  Michael Diaz,et al.  Measurement of relative metamnemonic accuracy. , 2008 .

[45]  Irwin Pollack,et al.  Sampling variability of the area under the ROC-curve and of d'e. , 1969 .

[46]  R. Lindsay,et al.  On Estimating the Diagnosticity of Eyewitness Nonidentifications , 1980 .

[47]  E. Heit,et al.  Traditional difference-score analyses of reasoning are flawed , 2014, Cognition.

[48]  Xavier Robin,et al.  pROC: an open-source package for R and S+ to analyze and compare ROC curves , 2011, BMC Bioinformatics.

[49]  N. Graham,et al.  Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation , 2002 .

[50]  Eyewitness Identification , 2014 .

[51]  Evan Heit,et al.  When more data steer us wrong: replications with the wrong dependent measure perpetuate erroneous conclusions , 2015, Psychonomic bulletin & review.

[52]  Nancy K. Steblay,et al.  Seventy-two tests of the sequential lineup superiority effect: A meta-analysis and policy discussion. , 2011 .

[53]  Johannes Fürnkranz,et al.  ROC ‘n’ Rule Learning—Towards a Better Understanding of Covering Algorithms , 2005, Machine Learning.

[54]  Caren M Rotello,et al.  Sources of bias in the Goodman-Kruskal gamma coefficient measure of association: implications for studies of metacognitive processes. , 2009, Journal of experimental psychology. Learning, memory, and cognition.

[55]  E. Heit,et al.  Assessing the belief bias effect with ROCs: it's a response bias effect. , 2010, Psychological review.