Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error

When making an inference or comparison with uncertain, noisy, or incomplete data, measurement error and confidence intervals can be as important for judgment as the actual mean values of different groups. These often misunderstood statistical quantities are frequently represented by bar charts with error bars. This paper investigates drawbacks with this standard encoding, and considers a set of alternatives designed to more effectively communicate the implications of mean and error data to a general audience, drawing from lessons learned from the use of visual statistics in the information visualization community. We present a series of crowd-sourced experiments that confirm that the encoding of mean and error significantly changes how viewers make decisions about uncertain data. Careful consideration of design tradeoffs in the visual presentation of data results in human reasoning that is more consistently aligned with statistical inferences. We suggest the use of gradient plots (which use transparency to encode uncertainty) and violin plots (which use width) as better alternatives for inferential tasks than bar charts with error bars.

[1]  M. Granger Morgan,et al.  Graphical Communication of Uncertain Quantities to Nontechnical People , 1987 .

[2]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[3]  F. Schmidt.  Statistical Significance Testing and Cumulative Knowledge in Psychology: Implications for Training of Researchers , 1996 .

[4]  Peter Kampstra,et al.  Beanplot: A Boxplot Alternative for Visual Comparison of Distributions , 2008 .

[5]  G. Cumming,et al.  Researchers misunderstand confidence intervals and standard error bars. , 2005, Psychological methods.

[6]  Nahum D. Gershon.  Visualization of an Imperfect World , 1998, IEEE Computer Graphics and Applications.

[7]  T. van der Geest,et al.  What is beautiful is useful - Visual appeal and expected information quality , 2009, 2009 IEEE International Professional Communication Conference.

[8]  Norm Goldstein.  The Associated Press stylebook and libel manual : including guidelines on photo captions, filing the wire, proofreaders' marks, copyright , 1992 .

[9]  Christopher H Jackson,et al.  Displaying Uncertainty With Shading , 2008 .

[10]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[11]  G. Cumming,et al.  Inference by eye: confidence intervals and how to read pictures of data. , 2005, The American psychologist.

[12]  Mark Gahegan,et al.  Visual Semiotics & Uncertainty Visualization: An Empirical Study , 2012, IEEE Transactions on Visualization and Computer Graphics.

[13]  Tobias Isenberg,et al.  Evaluating Sketchiness as a Visual Variable for the Depiction of Qualitative Uncertainty , 2012, IEEE Transactions on Visualization and Computer Graphics.

[14]  Lisa L. Harlow,et al.  Eight Common but False Objections to the Discontinuation of Significance Testing in the Analysis of Research Data , 2016 .

[15]  Joe Michael Kniss,et al.  Visualizing Summary Statistics and Uncertainty , 2010, Comput. Graph. Forum.

[16]  Cynthia A. Brewer,et al.  ColorBrewer.org: An Online Tool for Selecting Colour Schemes for Maps , 2003 .

[17]  Alan M. MacEachren,et al.  VISUALIZING UNCERTAIN INFORMATION , 1992 .

[18]  Michael Gleicher,et al.  Error Bars Considered Harmful , 2013 .

[19]  William S. Cleveland,et al.  A Color-Caused Optical Illusion on a Statistical Graph , 1983 .

[20]  David H. Laidlaw,et al.  Representing Uncertainty in Graph Edges: An Evaluation of Paired Visual Variables , 2015, IEEE Transactions on Visualization and Computer Graphics.

[21]  Ohad Inbar,et al.  Graphical representation of statistical information in situations of judgment and decision-making , 2007, ECCE '07.

[22]  Eytan Adar,et al.  Benefitting InfoVis with Visual Difficulties , 2011, IEEE Transactions on Visualization and Computer Graphics.

[23]  Thea van der Geest,et al.  What is beautiful is useful - Visual appeal and expected information quality , 2009 .

[24]  A. Tversky,et al.  Judgment under Uncertainty: Heuristics and Biases , 1974, Science.

[25]  Alexander Toet,et al.  The perception of visual uncertainty representation by non-experts , 2013 .

[26]  Douglas H. Johnson.  The Insignificance of Statistical Significance Testing , 1999 .

[27]  Pierre Dragicevic,et al.  Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing , 2012, IEEE Transactions on Visualization and Computer Graphics.

[28]  John T. Behrens,et al.  Box, Line, and Midgap Plots: Effects of Display Characteristics on the Accuracy and Bias of Estimates of Whisker Length , 1991 .

[29]  R. Nickerson,et al.  Null hypothesis significance testing: a review of an old and continuing controversy. , 2000, Psychological methods.

[30]  Susan Bell Trickett,et al.  Extracting Explicit and Implict Information from Complex Visualizations , 2002, Diagrams.

[31]  W. Cleveland,et al.  Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods , 1984 .

[32]  Steven Franconeri,et al.  Comparing averages in time series data , 2012, CHI.

[33]  Tobias Isenberg,et al.  Sketchy Rendering for Information Visualization , 2012, IEEE Transactions on Visualization and Computer Graphics.

[34]  Charles S. Myers,et al.  The Müller-Lyer illusion. , 2022 .

[35]  Alexander Toet,et al.  The Perception of Visual UncertaintyRepresentation by Non-Experts , 2014, IEEE Transactions on Visualization and Computer Graphics.

[36]  Brian J Scholl,et al.  Bar graphs depicting averages are perceptually misinterpreted: The within-the-bar bias , 2012, Psychonomic bulletin & review.

[37]  J. Hintze,et al.  Violin plots : A box plot-density trace synergism , 1998 .

[38]  J. G. Hollands,et al.  The visual communication of risk. , 1999, Journal of the National Cancer Institute. Monographs.

[39]  Norm Goldstein.  The Associated Press Stylebook and Libel Manual. Fully Revised and Updated. , 1994 .