Too good to be true: when overwhelming evidence fails to convince

Is it possible for a large sequence of measurements or observations, which support a hypothesis, to counterintuitively decrease our confidence? Can unanimous support be too good to be true? The assumption of independence is often made in good faith; however, rarely is consideration given to whether a systemic failure has occurred. Taking this into account can cause certainty in a hypothesis to decrease as the evidence for it becomes apparently stronger. We perform a probabilistic Bayesian analysis of this effect with examples based on (i) archaeological evidence, (ii) weighing of legal evidence and (iii) cryptographic primality testing. In this paper, we investigate the effects of small error rates in a set of measurements or observations. We find that even with very low systemic failure rates, high confidence is surprisingly difficult to achieve; in particular, we find that certain analyses of cryptographically important numerical tests are highly optimistic, underestimating their false-negative rate by as much as a factor of 280.

[1]  Isidore Epstein.  The Babylonian Talmud , 1978 .

[2]  Ariel Orda,et al.  Avoiding the Braess paradox in non-cooperative networks , 1999, Journal of Applied Probability.

[3]  Roy S. Malpass,et al.  Eyewitness identification: Lineup instructions and the absence of the offender. , 1981 .

[4]  Bruce Schneier,et al.  Cryptography Engineering - Design Principles and Practical Applications , 2010 .

[5]  Doreen Eichel,et al.  Data Analysis A Bayesian Tutorial , 2016 .

[6]  P. Landa.  Mechanism of stochastic resonance , 2004 .

[7]  Eitan Altman,et al.  Braess-like paradoxes in distributed computer systems , 2000, IEEE Trans. Autom. Control..

[8]  Dietrich Braess,et al.  Über ein Paradoxon aus der Verkehrsplanung , 1968, Unternehmensforschung.

[9]  Consequentiality and eyewitness person identification , 1994 .

[10]  Patrick Devlin.  Report to the Secretary of State for the Home Department of the Departmental Committee on Evidence of Identification in Criminal Cases , 1976 .

[11]  Derek Abbott,et al.  Brownian ratchets and Parrondo's games. , 2001, Chaos.

[12]  A. Franklin.  Ending the Mendel-Fisher Controversy , 2008 .

[13]  Jerker Denrell,et al.  Top performers are not the most impressive when extreme performance indicates unreliability , 2012, Proceedings of the National Academy of Sciences.

[14]  C. Pearce,et al.  Stochastic Resonance: From Suprathreshold Stochastic Resonance to Stochastic Signal Quantization , 2008 .

[15]  Derek Abbott,et al.  Game theory: Losing strategies can win by Parrondo's paradox , 1999, Nature.

[16]  Rory A. Fisher,et al.  Has Mendel's work been rediscovered? , 1936 .

[17]  Bruce D. Spencer,et al.  Estimating the Accuracy of Jury Verdicts , 2007 .

[18]  E. Olsson.  Bayesian epistemology , 2017 .

[19]  Chris Fallin,et al.  Flipping bits in memory without accessing them: An experimental study of DRAM disturbance errors , 2014, 2014 ACM/IEEE 41st International Symposium on Computer Architecture (ISCA).

[20]  Eduardo Pinheiro,et al.  DRAM errors in the wild: a large-scale field study , 2009, SIGMETRICS '09.

[21]  S. Ethier,et al.  Parrondo's paradox via redistribution of wealth , 2011, 1109.4454.

[22]  Derek Abbott,et al.  What Is Stochastic Resonance? Definitions, Misconceptions, Debates, and Its Relevance to Biology , 2009, PLoS Comput. Biol..

[23]  Derek Abbott,et al.  Asymmetry and disorder: A decade of Parrondo's Paradox , 2010 .

[24]  M. Wogalter,et al.  Suggestiveness in photospread line‐ups: Similarity induces distinctiveness , 1992 .

[25]  Derek Abbott,et al.  Quantum two- and three-person duels , 2004 .