Visual displays for communicating scientific uncertainty in influenza forecasts

We offer a general method for testing the usability of visual displays communicating scientific uncertainty, illustrated with publicly available results from CDC's influenza forecasts. The heavy toll of seasonal influenza has prompted major investments in improving these forecasts, making them a focus of machine learning research. However, little research has been devoted to how well users can understand and use these forecasts to inform decisions under uncertainty. Our approach extends psychological theory to experimental tasks posing hypothetical, but realistic decisions using alternative displays based on actual forecasts. Based on Tversky's theory of conceptual-spatial congruence, we predicted actual and perceived usability of four displays (bar chart, tree map, PDF, and 90% confidence interval). Participants (N = 301, recruited on Amazon MTurk) were randomly assigned to use one of four displays for four decision tasks, created to reflect our extension of the theory. We evaluated participants' comprehension, confidence, and judgments of perceived helpfulness, when the display and the decision were congruent or non-congruent. Participants had better comprehension with the most familiar display (bar chart), for all four decisions. However, they did not perceive that display as more helpful or have greater confidence in their responses to it. Participants who reported greater familiarity with a display performed more poorly, despite expressing greater confidence and rating it as more helpful. We discuss the need to evaluate performance, as well as ratings, and the opportunities to extend theoretical frameworks to specific contexts.