From Mice to Men - 24 Years of Evaluation in CHI

This paper analyzes trends in the approach to evaluation taken by CHI papers in the last 24 years. A set of papers was analyzed according to our schema for classifying type of evaluation. Our analysis traces papers’ trend in type and scope of evaluation. Findings include an increase in the proportion of papers that include evaluation, and a decrease in the median number of subjects in quantitative studies. We also critique the types of subjects, in particular an over reliance on students, and lack of appropriately gender balanced samples. We contextualize these findings in historical trends as we move from machines intended for the technical elite in laboratories to computers integrated into the daily life of everyone.

[1]  Steve Benford,et al.  The Frame of the Game: Blurring the Boundary between Fiction and Reality in Mobile Experiences , 2006, CHI 2006.

[2]  Anind K. Dey,et al.  From awareness to connectedness: the design and deployment of presence displays , 2006, CHI.

[3]  Wayne D. Gray,et al.  Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods , 1998, Hum. Comput. Interact..

[4]  Tracee Vetting Wolf,et al.  Dispelling "design" as the black art of CHI , 2006, CHI.

[5]  Robert J.K. Jacob.  Executable specifications for a human-computer interface , 1983, CHI '83.

[6]  P. Barnard,et al.  Design practice and interface usability: Evidence from interviews with designers , 1983, CHI '83.

[7]  Jakob Nielsen,et al.  Usability engineering at a discount , 1989 .

[8]  Kate Ehrlich,et al.  Tapping into tacit programming knowledge , 1982, CHI '82.

[9]  Paul Dourish,et al.  Implications for design , 2006, CHI.

[10]  Alan Morse,et al.  Some principles for the effective display of data , 1979, SIGGRAPH '79.

[11]  Steve Benford,et al.  The Frame of the Game: Blurring the Boundary between Fiction and Reality in Mobile Experiences , 2006, CHI 2006.

[12]  Helen Petrie,et al.  Remote usability evaluations With disabled people , 2006, CHI.

[13]  Duncan Rowland,et al.  Interweaving mobile games with everyday life , 2006, CHI.

[14]  S. Turkle.  Computational reticence: why women fear the intimate machine , 2004 .

[15]  Albrecht Schmidt,et al.  The drift table: designing for ludic engagement , 2004, CHI EA '04.

[16]  Robert J. Moore,et al.  "Alone together?": exploring the social dynamics of massively multiplayer online games , 2006, CHI.

[17]  Avi Parush.  Toward a common ground: practice and research in HCI , 2006, INTR.

[18]  Allan G. Haggett,et al.  Naive user behavior in a restricted interactive command environment (abstract only) , 1982 .

[19]  Jonathan Grudin.  Is HCI homeless?: in search of inter-disciplinary status , 2006, INTR.

[20]  Marilyn Mantei,et al.  Autobiography of a first-time discretionary microcomputer user , 1983, CHI '83.

[21]  Marc Hassenzahl,et al.  Funology: designing enjoyment , 2002, CHI Extended Abstracts.

[22]  Phoebe Sengers,et al.  Staying open to interpretation: engaging multiple meanings in design and evaluation , 2006, DIS '06.

[23]  Albert T. Corbett,et al.  Instructional interventions in computer-based tutoring: differential impact on learning time and accuracy , 2000, CHI.

[24]  Regan L. Mandryk,et al.  A continuous and objective evaluation of emotional experience with interactive play environments , 2006, CHI.

[25]  Jonathan Arnowitz,et al.  CHI and the practitioner dilemma , 2005, INTR.

[26]  Wendy E. Mackay.  The CHI conference review process: writing and interpreting paper reviews , 1998, CHI Conference Summary.

[27]  Peter Calingaert,et al.  System performance evaluation , 1967, Commun. ACM.

[28]  David R. Cheriton.  Man-machine interface design for timesharing systems , 1976, ACM '76.

[29]  Leysia Palen,et al.  Participatory design in emergency medical service: designing for future practice , 2006, CHI.

[30]  William Newman,et al.  A preliminary analysis of the products of HCI research, using pro forma abstracts , 1994, CHI '94.

[31]  Jeff A. Johnson,et al.  User preferences for task-specific vs. generic application software , 1994, CHI '94.

[32]  Alison Kidd,et al.  The marks are on the knowledge worker , 1994, CHI '94.

[33]  C. Egido,et al.  Pictures and category labels as navigational aids for catalog browsing , 1988, CHI '88.

[34]  I. H. Witten,et al.  How users repeat their actions on computers: principles for design of history mechanisms , 1988, CHI '88.

[35]  Jonathan Grudin.  Why CHI fragmented , 2005, CHI EA '05.

[37]  Wendie Wulff,et al.  An assessment of HCI: issues and implications , 1990, SGCH.

[38]  M. Rosenzweig,et al.  Species diversity in space and time: To a mouse , 1995 .

[39]  John R. Anderson,et al.  Intelligent gaze-added interfaces , 2000, CHI.