Is once enough?: on the extent and content of replications in human-computer interaction

A replication is an attempt to confirm an earlier study's findings. It is often claimed that research in Human-Computer Interaction (HCI) contains too few replications. To investigate this claim we examined four publication outlets (891 papers) and found 3% attempting replication of an earlier result. The replications typically confirmed earlier findings, but treated replication as a confirm/not-confirm decision, rarely analyzing effect sizes or comparing in depth to the replicated paper. When asked, most authors agreed that their studies were replications, but rarely planned them as such. Many non-replication studies could have corroborated earlier work if they had analyzed data differently or used minimal effort to collect extra data. We discuss what these results mean to HCI, including how reporting of studies could be improved and how conferences/journals may change author instructions to get more replications.

[1]  Catherine C. Marshall,et al.  Designing Qualitative Research , 1996 .

[2]  Vincent Koenig,et al.  Replicating an International Survey on User Experience: Challenges, Successes and Limitations , 2013, RepliCHI.

[3]  Eun-Ju Lee,et al.  Flattery may get computers somewhere, sometimes: The moderating role of output modality, computer gender, and user gender , 2008, Int. J. Hum. Comput. Stud..

[4]  James W. Neuliep,et al.  Everyone was wrong: There are lots of replications out there. , 1993 .

[5]  Joanna McGrenere,et al.  Steadied-bubbles: combining techniques to address pen-based pointing errors for younger and older adults , 2010, CHI.

[6]  Poika Isokoski,et al.  EdgeWrite with integrated corner sequence help , 2008, CHI.

[7]  C. Begley,et al.  Drug development: Raise standards for preclinical cancer research , 2012, Nature.

[8]  Dan Morris,et al.  SearchBar: a search-centric web history for task resumption and information re-finding , 2008, CHI.

[9]  Joanna McGrenere,et al.  Impact of screen size on performance, awareness, and user satisfaction with adaptive graphical user interfaces , 2008, CHI.

[10]  Clifford Nass,et al.  Driver safety and information from afar: An experimental driving simulator study of wireless vs. in-car information services , 2008, Int. J. Hum. Comput. Stud..

[11]  Olivier Chapuis,et al.  High-precision magnification lenses , 2010, CHI.

[12]  C. Hendrick,et al.  Replications, strict replications, and conceptual replications: Are they important? , 1990 .

[13]  Daniel Vogel,et al.  Shift: a technique for operating pen-based interfaces using touch , 2007, CHI.

[14]  Uwe Flick,et al.  Designing Qualitative Research , 2008 .

[15]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[16]  Carl Gutwin,et al.  Targeting across displayless space , 2008, CHI.

[17]  Eun-Ju Lee,et al.  I like you, but I won't listen to you: Effects of rationality on affective and behavioral responses to computers that flatter , 2009, Int. J. Hum. Comput. Stud..

[18]  Desney S. Tan,et al.  Feasibility and pragmatics of classifying working memory load with an electroencephalograph , 2008, CHI.

[19]  Louis M. Gomez,et al.  Formative design evaluation of superbook , 1989, TOIS.

[20]  Robert Rosenthal,et al.  Replication in behavioral research. , 1990 .

[21]  Benjamin B. Bederson,et al.  One-handed touchscreen input for legacy applications , 2008, CHI.

[22]  Emmanuel Pietriga,et al.  Sigma lenses: focus-context transitions combining space, time and translucence , 2008, CHI.

[23]  T. Sterling.  Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa , 1959 .

[24]  Steven K. Feiner,et al.  Rubbing and tapping for precise and rapid selection on touch-screen displays , 2008, CHI.

[25]  Daniel Vogel,et al.  The effect of spring stiffness and control gain with an elastic rate control pointing device , 2008, CHI.

[26]  Jeffrey Heer,et al.  Crowdsourcing graphical perception: using mechanical turk to assess visualization design , 2010, CHI.

[27]  Leonid Kruglyak,et al.  Absence of Detectable Arsenate in DNA from Arsenate-Grown GFAJ-1 Cells , 2012, Science.

[28]  Antonella De Angeli,et al.  Framing the user experience: information biases on website quality judgement , 2008, CHI.

[29]  John R. Huizenga,et al.  Cold fusion: The scientific fiasco of the century , 1992 .

[30]  Olivier Chapuis,et al.  DynaSpot: speed-dependent area cursor , 2009, CHI.

[31]  Keith S. Jones,et al.  An Investigation of the Prevalence of Replication Research in Human Factors , 2010, Hum. Factors.

[32]  William Buxton,et al.  Usability evaluation considered harmful (some of the time) , 2008, CHI.

[33]  Mark W. Newman,et al.  Escape: a target selection technique using visually-cued gestures , 2008, CHI.

[34]  J Hilliard,et al.  Again and Again and Again , 2005 .

[35]  John E. Hunter,et al.  Methods of Meta-Analysis , 1989 .

[36]  J. McGrath.  Methodology matters: doing research in the behavioral and social sciences , 1995 .

[37]  J. Ioannidis.  Why Most Published Research Findings Are False , 2005, PLoS medicine.

[38]  Stephen A. Brewster,et al.  Investigating the effectiveness of tactile feedback for mobile touchscreens , 2008, CHI.

[39]  Ravin Balakrishnan,et al.  Evaluating tactile feedback and direct vs. indirect stylus input in pointing and crossing selection tasks , 2008, CHI.

[40]  J. Scott Armstrong,et al.  Replication research's disturbing trend , 2007 .

[41]  Shumin Zhai,et al.  Human on-line response to target expansion , 2003, CHI '03.

[42]  Michael S. Bernstein,et al.  RepliCHI - CHI should be replicating and validating results more: discuss , 2011, CHI Extended Abstracts.

[43]  J. Mackie,et al.  The Conduct of Inquiry: Methodology for Behavioural Science , 1965 .

[44]  Arthur C. Graesser,et al.  Toward Spoken Human–Computer Tutorial Dialogues , 2010, Hum. Comput. Interact..

[45]  Bonnie E. John.  Avoiding "It's JUST a Replication" , 2013, RepliCHI.

[46]  Kasper Hornbæk,et al.  Some Whys and Hows of Experiments in Human-Computer Interaction , 2013, Found. Trends Hum. Comput. Interact..

[47]  William Newman,et al.  A preliminary analysis of the products of HCI research, using pro forma abstracts , 1994, CHI '94.

[48]  L. J. Chase,et al.  REPLICATION IN EXPERIMENTAL COMMUNICATION RESEARCH: AN ANALYSIS , 1979 .

[49]  François Guimbretière,et al.  Relative role of merging and two-handed operation on command selection speed , 2008, Int. J. Hum. Comput. Stud..

[50]  Eric W. K. Tsang,et al.  Replication and Theory Development in Organizational Science: A Critical Realist Perspective , 1999 .

[51]  Philip Tuddenham,et al.  Graspables revisited: multi-touch vs. tangible input for tabletop displays in acquisition and manipulation tasks , 2010, CHI.

[52]  John F. Hughes,et al.  Indirect mappings of multi-touch input using one and two hands , 2008, CHI.