How to measure metacognition

The ability to recognize one's own successful cognitive processing, in e.g., perceptual or memory tasks, is often referred to as metacognition. How should we quantitatively measure such ability? Here we focus on a class of measures that assess the correspondence between trial-by-trial accuracy and one's own confidence. In general, for healthy subjects endowed with metacognitive sensitivity, when one is confident, one is more likely to be correct. Thus, the degree of association between accuracy and confidence can be taken as a quantitative measure of metacognition. However, many studies use a statistical correlation coefficient (e.g., Pearson's r) or its variant to assess this degree of association, and such measures are susceptible to undesirable influences from factors such as response biases. Here we review other measures based on signal detection theory and receiver operating characteristics (ROC) analysis that are “bias free,” and relate these quantities to the calibration and discrimination measures developed in the probability estimation literature. We go on to distinguish between the related concepts of metacognitive bias (a difference in subjective confidence despite basic task performance remaining constant), metacognitive sensitivity (how good one is at distinguishing between one's own correct and incorrect judgments) and metacognitive efficiency (a subject's level of metacognitive sensitivity given a certain level of task performance). Finally, we discuss how these three concepts pose interesting questions for the study of metacognition and conscious awareness.

[1]  J. Jastrow,et al.  On small differences in sensation , 1884 .

[2]  V. Henmon,et al.  The relation of the time of a judgment to its accuracy. , 1911 .

[3]  G. Brier VERIFICATION OF FORECASTS EXPRESSED IN TERMS OF PROBABILITY , 1950 .

[4]  T. Birdsall,et al.  Two Types of ROC Curves and Definitions of Parameters , 1959 .

[5]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[6]  Joseph Lebacqz,et al.  SUBJECTIVE AND OBJECTIVE , 1967 .

[7]  EVALUATION OF A , 1972 .

[8]  A. H. Murphy A New Vector Partition of the Probability Score , 1973 .

[9]  L Weiskrantz,et al.  Visual capacity in the hemianopic field following a restricted occipital ablation. , 1974, Brain : a journal of neurology.

[10]  C. Thronesbery,et al.  Metamemory through the Adult Life Span. , 1979 .

[11]  L. A. Goodman,et al.  Measures of association for cross classifications , 1979 .

[12]  William R. Ferrell,et al.  A model of calibration for subjective probabilities , 1980 .

[13]  B. Fischhoff,et al.  Calibration of probabilities: the state of the art to 1980 , 1982 .

[14]  T. O. Nelson,et al.  A comparison of current measures of the accuracy of feeling-of-knowing predictions. , 1984, Psychological bulletin.

[15]  D. C. Howell Statistical Methods for Psychology , 1987 .

[16]  T. O. Nelson Metamemory: A Theoretical Framework and New Findings , 1990 .

[17]  T. O. Nelson,et al.  When People's Judgments of Learning (JOLs) are Extremely Accurate at Predicting Subsequent Recall: The “Delayed-JOL Effect” , 1991 .

[18]  G. Keren Calibration and probability judgements: Conceptual and methodological issues , 1991 .

[19]  Ilan Yaniv,et al.  Measures of Discrimination Skill in Probabilistic Judgment , 1991 .

[20]  J. Metcalfe,et al.  Metacognition : knowing about knowing , 1994 .

[21]  Natika Newton,et al.  Metacognition and consciousness , 1995 .

[22]  Jochen Braun,et al.  Blindsight in normal observers , 1995, Nature.

[23]  T. O. Nelson,et al.  Measuring ordinal association in situations that contain tied scores. , 1996, Psychological bulletin.

[24]  M. Morgan,et al.  Blindsight in normal subjects? , 1997, Nature.

[25]  Nigel Harvey,et al.  Confidence in judgment , 1997, Trends in Cognitive Sciences.

[26]  J. Kruger,et al.  Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. , 1999, Journal of personality and social psychology.

[27]  M. Isingrini,et al.  Aging, episodic memory feeling-of-knowing, and frontal functioning. , 2000, Neuropsychology.

[28]  R. Hampton Rhesus monkeys know when they remember , 2001, Proceedings of the National Academy of Sciences of the United States of America.

[29]  H. Pashler,et al.  Confidence and Accuracy of Near-Threshold Discrimination Responses , 2001, Consciousness and Cognition.

[30]  Susan J. Galvin,et al.  Type 2 tasks in the theory of signal detectability: Discrimination between correct and incorrect decisions , 2003, Psychonomic bulletin & review.

[31]  R. Passingham,et al.  Relative blindsight in normal observers and the neural correlate of visual consciousness , 2006, Proceedings of the National Academy of Sciences.

[32]  Sterling C. Johnson,et al.  Neural correlates of self-evaluative accuracy after traumatic brain injury , 2006, Neuropsychologia.

[33]  P. Higham,et al.  No special K! A signal detection framework for the strategic regulation of memory accuracy. , 2007, Journal of experimental psychology. General.

[34]  P. Azzopardi,et al.  Evaluation of a 'bias-free' measure of awareness. , 2007, Spatial vision.

[35]  Lisa K. Son,et al.  Transfer of Metacognitive Skills and Hint Seeking in Monkeys , 2007 .

[36]  Michael Diaz,et al.  Measurement of relative metamnemonic accuracy. , 2008 .

[37]  Z. Dienes Subjective measures of unconscious knowledge. , 2008, Progress in brain research.

[38]  D. Moore,et al.  The trouble with overconfidence. , 2008, Psychological review.

[39]  Caren M Rotello,et al.  Sources of bias in the Goodman-Kruskal gamma coefficient measure of association: implications for studies of metacognitive processes. , 2009, Journal of experimental psychology. Learning, memory, and cognition.

[40]  Davide Bruno,et al.  Investigating strength and frequency effects in recognition memory using type-2 signal detection theory. , 2009, Journal of experimental psychology. Learning, memory, and cognition.

[41]  R. Passingham,et al.  Theta-burst transcranial magnetic stimulation to the prefrontal cortex impairs metacognitive visual awareness , 2010, Cognitive neuroscience.

[42]  Axel Cleeremans,et al.  Measuring consciousness: Is one measure better than the other? , 2010, Consciousness and Cognition.

[43]  Geraint Rees,et al.  Relating Introspective Accuracy to Individual Differences in Brain Structure , 2010, Science.

[44]  J. Vergnaud,et al.  Subjective beliefs formation and elicitation rules : experimental evidence , 2010 .

[45]  I. Jolliffe,et al.  Forecast verification : a practitioner's guide in atmospheric science , 2011 .

[46]  Matthew Davidson,et al.  Awareness-related activity in prefrontal and parietal cortices in blindsight reflects more than superior visual performance , 2011, NeuroImage.

[47]  Matthew G. Rhodes,et al.  The influence of delaying judgments of learning on metacognitive accuracy: a meta-analytic review. , 2011, Psychological bulletin.

[48]  Caspar M. Schwiedrzik,et al.  Subjective and objective learning effects dissociate in space and in time , 2011, Proceedings of the National Academy of Sciences.

[49]  Paul G. Middlebrooks,et al.  Metacognition in monkeys during an oculomotor task. , 2011, Journal of experimental psychology. Learning, memory, and cognition.

[50]  Sandy L. Zabell,et al.  The Subjective and the Objective , 2011 .

[51]  Hakwan C. Lau,et al.  Are we studying consciousness yet , 2012 .

[52]  Johann A. Makowsky,et al.  A Computational Framework for the Study of Partition Functions and Graph Polynomials , 2012, 2012 14th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing.

[53]  Thomas S Wallsten,et al.  A stochastic detection and retrieval model for the study of metacognition. , 2012, Psychological review.

[54]  R. Dolan,et al.  The neural basis of metacognitive ability , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[55]  Adam Kepecs,et al.  A computational framework for the study of confidence in humans and animals , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[56]  A. David,et al.  Failures of metacognition and lack of insight in neuropsychiatric disorders , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[57]  H. Lau,et al.  A signal detection theoretic approach for estimating metacognitive sensitivity from confidence ratings , 2012, Consciousness and Cognition.

[58]  Stanislas Dehaene,et al.  Distinct Brain Mechanisms for Conscious versus Subliminal Error Detection , 2022 .

[59]  Krzysztof J. Gorgolewski,et al.  Medial and Lateral Networks in Anterior Prefrontal Cortex Support Metacognitive Ability for Memory and Perception , 2013, The Journal of Neuroscience.

[60]  Mark D'Esposito,et al.  Disruption of Dorsolateral But Not Ventrolateral Prefrontal Cortex Improves Unconscious Perceptual Memories , 2013, The Journal of Neuroscience.

[61]  Floris P. de Lange,et al.  Anatomical Coupling between Distinct Metacognitive Systems for Memory and Visual Perception , 2013, The Journal of Neuroscience.

[62]  Zoltan Dienes,et al.  Measures of metacognition on signal-detection theoretic models. , 2013, Psychological methods.

[63]  Geraint Rees,et al.  The development of metacognitive ability in adolescence , 2013, Consciousness and Cognition.

[64]  R. Dolan,et al.  Confidence in value-based choice , 2012, Nature Neuroscience.

[65]  Pascal Mamassian,et al.  Does Confidence Use a Common Currency Across Two Visual Tasks? , 2014, Psychological science.

[66]  Hakwan Lau,et al.  Signal Detection Theory Analysis of Type 1 and Type 2 Data: Meta-d′, Response-Specific Meta-d′, and the Unequal Variance SDT Model , 2014 .