Quantifying Stimulus Discriminability: A Comparison of Information Theory and Ideal Observer Analysis

Performance in sensory discrimination tasks is commonly quantified using either information theory or ideal observer analysis. These two quantitative frameworks are often assumed to be equivalent. For example, higher mutual information is said to correspond to improved performance of an ideal observer in a stimulus estimation task. To the contrary, drawing on and extending previous results, we show that five information-theoretic quantities (entropy, response-conditional entropy, specific information, equivocation, and mutual information) violate this assumption. More positively, we show how these information measures can be used to calculate upper and lower bounds on ideal observer performance, and vice versa. The results show that the mathematical resources of ideal observer analysis are preferable to information theory for evaluating performance in a stimulus discrimination task. We also discuss the applicability of information theory to questions that ideal observer analysis cannot address.

[1]  Terry J. Wagner,et al.  Some remarks concerning uncertainty and the probability of error (Corresp.) , 1965, IEEE Trans. Inf. Theory.

[2]  M R DeWeese,et al.  How to measure the information gained from one symbol. , 1999, Network.

[3]  Forbes Ad,et al.  Classification-algorithm evaluation: five performance measures based on confusion matrices. , 1995 .

[4]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[5]  Jovan Dj. Golic.  On the relationship between the information measures and the Bayes probability of error , 1987, IEEE Trans. Inf. Theory.

[6]  Neri Merhav,et al.  Relations between entropy and error probability , 1994, IEEE Trans. Inf. Theory.

[7]  A. Gelperin,et al.  Olfactory neuronal dynamics in behaving animals. , 2006, Seminars in cell & developmental biology.

[8]  W. McCulloch,et al.  The limiting information capacity of a neuronal link , 1952 .

[9]  J P Miller,et al.  Representation of sensory information in the cricket cercal sensory system. II. Information theoretic calculation of system accuracy and optimal tuning-curve widths of four primary interneurons. , 1991, Journal of neurophysiology.

[10]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[11]  David C. Knill,et al.  Object classification for human and ideal observers , 1995, Vision Research.

[12]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[13]  T. Albright,et al.  Gauging sensory representations in the brain , 1999, Trends in Neurosciences.

[14]  Carl de Boor,et al.  A Practical Guide to Splines , 1978, Applied Mathematical Sciences.

[15]  M. Diamond,et al.  Whisker Vibration Information Carried by Rat Barrel Cortex Neurons , 2004, The Journal of Neuroscience.

[16]  Eilon Vaadia,et al.  Learning-Induced Improvement in Encoding and Decoding of Specific Movement Directions by Neurons in the Primary Motor Cortex , 2004, PLoS biology.

[17]  J D Victor,et al.  Temporal aspects of neural coding in the retina and lateral geniculate. , 1999, Network.

[18]  A Treves,et al.  On the perceptual structure of face space. , 1997, Bio Systems.

[19]  Michael J. Berry,et al.  Synergy, Redundancy, and Independence in Population Codes , 2003, The Journal of Neuroscience.

[20]  Samuel J. Dwyer,et al.  Uncertainty and the probability of error (Corresp.) , 1968, IEEE Trans. Inf. Theory.

[21]  William Bialek,et al.  Synergy in a Neural Code , 2000, Neural Computation.

[22]  J. Movshon,et al.  The analysis of visual motion: a comparison of neuronal and psychophysical performance , 1992, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[23]  W. Geisler.  Ideal Observer Analysis , 2002 .

[24]  Liam Paninski,et al.  Estimating entropy on m bins given fewer than m samples , 2004, IEEE Transactions on Information Theory.

[25]  P. Latham,et al.  Synergy, Redundancy, and Independence in Population Codes, Revisited , 2005, The Journal of Neuroscience.

[26]  A. Forbes,et al.  Classification-algorithm evaluation: Five performance measures based onconfusion matrices , 2005, Journal of Clinical Monitoring.

[27]  John S. Kauer,et al.  Characterizing complex chemosensors: information-theoretic analysis of olfactory systems , 1999, Trends in Neurosciences.

[28]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[29]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[30]  V. A. Kovalevsky,et al.  Character readers and pattern recognition , 1968 .

[31]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[32]  William Bialek,et al.  Spikes: Exploring the Neural Code , 1996 .

[33]  William Bialek,et al.  Entropy and Information in Neural Spike Trains , 1996, cond-mat/9603127.

[34]  M. Diamond,et al.  Population coding in somatosensory cortex , 2002, Current Opinion in Neurobiology.

[35]  W. Pitts,et al.  What the Frog's Eye Tells the Frog's Brain , 1959, Proceedings of the IRE.

[36]  Wilson S. Geisler,et al.  Ideal observer theory in psychophysics and physiology , 1989 .

[37]  Alon Orlitsky,et al.  Always Good Turing: Asymptotically Optimal Probability Estimation , 2003, Science.

[38]  D. M. Green,et al.  Signal detection theory and psychophysics , 1966 .

[39]  L. Goddard.  Information Theory , 1962, Nature.

[40]  A. U.S.,et al.  Predictability , Complexity , and Learning , 2002 .

[41]  Michael J. Berry,et al.  Refractoriness and Neural Precision , 1997, The Journal of Neuroscience.

[42]  D. Perrett,et al.  The `Ideal Homunculus': decoding neural population signals , 1998, Trends in Neurosciences.

[43]  C. R. Deboor,et al.  A practical guide to splines , 1978 .

[44]  C. Gilbert,et al.  Perceptual learning and top-down influences in primary visual cortex , 2004, Nature Neuroscience.

[45]  S. Panzeri,et al.  An exact method to quantify the information transmitted by different mechanisms of correlational coding. , 2003, Network.