HCI... not as it should be: inferential statistics in HCI research

This paper surveys the use of inferential statistics over the last two BCS HCI conferences and the last year (2006) of two leading HCI journals. Of the 80 papers covered, 41 used some form of inferential statistics. However, all but one had some form of problem of reporting or analysis that undermined the value or the validity of the statistical testing and hence the research findings. This paper discusses the implications of such widespread issues for HCI research and considers approaches for improving the use of statistics in HCI.

[1]  R. Abelson.  Statistics As Principled Argument , 1995 .

[2]  Jacob Cohen.  The earth is round (p < .05) , 1994 .

[3]  J. Algina,et al.  Univariate and Multivariate Omnibus Hypothesis Tests Selected to Control Type I Error Rates When Population Variances Are Not Necessarily Equal , 1996 .

[4]  Robert Rosenthal,et al.  Evaluating Theories Efficiently: The Nuts and Bolts of Contrast Analysis , 2003 .

[5]  J. Tukey.  The Philosophy of Multiple Comparisons , 1991 .

[6]  Peter C. Wright,et al.  Funology: from usability to enjoyment , 2005 .

[7]  Jan Gulliksen,et al.  People and Computers XIX - The Bigger Picture: Proceedings of HCI 2005 (BCS Conference) , 2005 .

[8]  G. Box.  Some Theorems on Quadratic Forms Applied in the Study of Analysis of Variance Problems, I. Effect of Inequality of Variance in the One-Way Classification , 1954 .

[9]  M. Smithson.  Confidence Intervals (7th ed) , 2003 .

[10]  Carl J. Huberty,et al.  Statistical Practices of Educational Researchers: An Analysis of their ANOVA, MANOVA, and ANCOVA Analyses , 1998 .

[11]  Derek,et al.  Statistics Without Tears , 1981 .

[12]  Harold W. Thimbleby,et al.  SUPPORTING DIVERSE HCI RESEARCH , 2004 .

[13]  G. Gigerenzer.  Mindless statistics , 2004 .

[14]  R. Rosenthal,et al.  If you're Looking at the Cell Means, You're Not Looking at Only the Interaction (Unless All Main Effects Are Zero) , 1991 .

[15]  R. Blair,et al.  A more realistic look at the robustness and Type II error properties of the t test to departures from population normality. , 1992 .