How to Run Experiments: A Practical Guide to Research with Human Participants

and Preface .................................................................................................................................. 2 Acknowledgements ................................................................................................................................... 4 Blurbs ........................................................................................................................................................ 5 1 Overview of the Research Process ........................................................................................... 9 1.1 Overview ............................................................................................................................................ 9 1.2 Overview of the research process ..................................................................................................... 12 1.3 Overview of the running examples ................................................................................................... 17 1.4 Further readings ................................................................................................................................ 20 1.5 Questions .......................................................................................................................................... 21 Summary questions ............................................................................................................................. 21 Thought questions ............................................................................................................................... 21 2 Preparation for Running Experiments .................................................................................... 23 2.1 Literature in the area ......................................................................................................................... 24 2.2 Choice of a term: Participants or subjects ........................................................................................ 24 2.3 Recruiting participants ...................................................................................................................... 25 2.4 Subject pools and class-based participation ..................................................................................... 27 2.5 Care, control, use, and maintenance of apparatus ............................................................................ 28 2.5.1 Experimental software ............................................................................................................... 28 2.5.2 E-Prime ...................................................................................................................................... 29 2.5.3 Keystroke loggers ...................................................................................................................... 29 2.5.4 Eyetrackers ................................................................................................................................ 31 2.6 The testing facility ............................................................................................................................ 31 2.7 Choice of dependent measures: Performance, time, actions, errors, verbal protocol analysis, and other measures ................................................................................................................... 32 2.7.1 Types of dependent measures .................................................................................................... 33 2.7.2 Levels of measurement .............................................................................................................. 35 2.7.3 Scales of measurement .............................................................................................................. 36 2.8 Plan data collection with analysis in mind ....................................................................................... 37 2.9 Run analyses with pilot data ............................................................................................................. 38 2.10 Institutional Review Board (IRB) .................................................................................................. 38 2.11 What needs IRB approval? ............................................................................................................. 39 2.13 Preparing an IRB submission ......................................................................................................... 41 2.14 Writing about your experiment before running .............................................................................. 42 2.15 Preparing to run the low vision HCI study ..................................................................................... 42 2.16 Preparing to run the HRI study ....................................................................................................... 45 2.17 Conclusion ...................................................................................................................................... 46 2.18 Further readings .............................................................................................................................. 46 2.19 Questions ........................................................................................................................................ 47 Summary questions ............................................................................................................................. 47 Thought questions ............................................................................................................................... 47 3 Potential Ethical Problems ..................................................................................................... 48 3.1 Preamble: A simple study that hurt somebody ................................................................................ 48 3.2 The history and role of ethics reviews .............................................................................................. 49 3.3 Recruiting subjects ........................................................................................................................... 49 3.4 Coercion of participants ................................................................................................................... 50 3.5 Risks, costs, and benefits of participation ........................................................................................ 50 How to run experiments: A practical guide 7 3.6 Sensitive data .................................................................................................................................... 51 3.7 Plagiarism ......................................................................................................................................... 53 3.8 Fraud ................................................................................................................................................. 53 3.9 Conflicts of interest .......................................................................................................................... 54 3.10 Authorship and data ownership ...................................................................................................... 54 3.11 Potential ethical problems in the low vision HCI study ................................................................. 55 3.12 Potential ethical problems in the multilingual fonts study ............................................................. 56 3.13 Conclusion ...................................................................................................................................... 59 3.14 Further readings .............................................................................................................................. 59 3.15 Questions ........................................................................................................................................ 59 Summary questions ............................................................................................................................. 59 Thought questions ............................................................................................................................... 60 4 Risks to Validity to Avoid While Running an Experiment .................................................... 61 4.1 Validity defined: Surface, internal, and external .............................................................................. 61 4.2 Risks to internal validity ................................................................................................................... 63 4.2.1 Power: How many participants? ................................................................................................ 63 4.2.2 Experimenter effects .................................................................................................................. 65 4.2.3 Participant effects ...................................................................................................................... 66 4.2.4 Demand characteristics .............................................................................................................. 66 4.2.4 Randomization and counterbalancing ....................................................................................... 66 4.2.5 Abandoning the task .................................................................................................................. 68 4.3 Risks to external validity .................................................................................................................. 68 4.3.1 Task fidelity ............................................................................................................................... 68 4.3.2 Representativeness of your sample .

[1]  James E. Sheedy,et al.  The Effects of Visual Display Distance on Eye Accommodation, Head Posture, and Vision and Neck Symptoms , 2007, Hum. Factors.

[2]  L. Reder,et al.  The Strategy-Specific Nature of Improvement: The Power Law Applies by Strategy Within Task , 1998 .

[3]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[4]  Gary Jones,et al.  Using a Cognitive Architecture to Examine what Develops , 2000, Psychological science.

[5]  Dario D. Salvucci.  An integrated model of eye movements and visual encoding , 2001, Cognitive Systems Research.

[6]  Jong Wook Kim.  Procedural skills: From learning to forgetting , 2008 .

[7]  Katelyn Y. A. McKenna,et al.  Oxford Handbook of Internet Psychology , 2007 .

[8]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[9]  Mandeep K. Dhami,et al.  The role of representative design in an ecological approach to cognition. , 2004, Psychological bulletin.

[10]  Jakob Nielsen,et al.  Usability laboratories , 1994, Behavior and Information Technology.

[11]  Gavan Lintern,et al.  Simulator Design and Instructional Features for Air-to-Ground Attack: A Transfer Study , 1989 .

[12]  Hans Spada,et al.  A Cognitive Model of Agents in a Commons Dilemma , 1997 .

[13]  Penelope M. Sanderson,et al.  Exploratory sequential data analysis: foundations , 1994 .

[14]  Mary Beth Rosson,et al.  Usability Engineering: Scenario-based Development of Human-Computer Interaction , 2001 .

[15]  Urmila Kukreja,et al.  RUI: Recording user input from interfaces under Windows and Mac OS X , 2006, Behavior research methods.

[16]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[17]  I. Bailey,et al.  Visual Factors and Orientation‐Mobility Performance , 1982, American journal of optometry and physiological optics.

[18]  Jong W. Kim.  Investigation of Procedural Skills Degradation from Different Modalities , 2007 .

[19]  E. R. Crossman.  A THEORY OF THE ACQUISITION OF SPEED-SKILL∗ , 1959 .

[20]  Estes Wk.  The problem of inference from curves based on group data. , 1956 .

[21]  G. Fishman.  When your eyes have a wet nose: the evolution of the use of guide dogs and establishing the seeing eye. , 2003, Survey of ophthalmology.

[22]  Robin R. Murphy,et al.  Review of Human Studies Methods in HRI and Recommendations , 2010, Int. J. Soc. Robotics.

[23]  Frank E. Ritter,et al.  Automatically recording keystrokes in public clusters with RUI: issues and sample answers , 2007 .

[24]  Frank E. Ritter,et al.  Including a Model of Visual Processing With a Cognitive Architecture to Model a Simple Teleoperation Task , 2007 .

[25]  Daniel N. Cassenti,et al.  Intentional control of event counting. , 2004, Journal of experimental psychology. Learning, memory, and cognition.

[26]  E Digiusto.  Equity in authorship: a strategy for assigning credit when publishing. , 1994, Social science & medicine.

[27]  A. Heathcote,et al.  Averaging learning curves across and within participants , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[28]  Hermann Ebbinghaus,et al.  Memory: a contribution to experimental psychology. , 1987, Annals of neurosciences.

[29]  Mercer Jennifer Ann,et al.  PUBLICATION manual of the American Psychological Association. , 1952, Psychological bulletin.

[30]  Margaret J. Robertson,et al.  Design and Analysis of Experiments , 2006, Handbook of statistics.

[31]  John R. Anderson,et al.  Eye Movements Do Not Reflect Retrieval Processes , 2004, Psychological science.

[32]  Arthur F Kramer,et al.  Transfer of computer-based training to simulated driving in older adults. , 2009, Applied ergonomics.

[33]  William J. Ray,et al.  Methods Toward a Science of Behavior and Experience , 1981 .

[34]  Wayne D. Gray,et al.  Argus: A suite of tools for research in complex cognition , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[35]  D. C. Howell.  Fundamental Statistics for the Behavioral Sciences , 1985 .

[36]  Walter Schneider,et al.  STEP—A System for Teaching Experimental Psychology using E-Prime , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[37]  D. Campbell,et al.  EXPERIMENTAL AND QUASI-EXPERIMENT Al DESIGNS FOR RESEARCH , 2012 .

[38]  G. Vossel,et al.  Finally, After Two Decades , 2001 .

[39]  Frank E. Ritter,et al.  Using Multidisciplinary Expert Evaluations to Test and Improve Cognitive Model Interfaces , 2002 .

[40]  Kurt VanLehn.  Getting out of order: Avoiding lesson effects through instruction. , 2007 .

[41]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[42]  Allen Newell,et al.  Learning by chunking: a production system model of practice , 1987 .

[43]  Richard W. Pew,et al.  Human-system integration in the system development process : a new look , 2007 .

[44]  Henry L. Roediger.  What Should They Be Called , 2011 .

[45]  R. Thouless.  Experimental Psychology , 1939, Nature.

[46]  Robert Wall Emerson,et al.  Drop-off Detection with the Long Cane: Effects of Different Cane Techniques on Performance , 2009, Journal of visual impairment & blindness.

[47]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[48]  Karen A. F. Copeland.  Design and Analysis of Experiments, 5th Ed. , 2001 .

[49]  D. Yeager,et al.  Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples , 2011 .

[50]  Andrew Heathcote,et al.  Repealing the power law: the case for an exponential law of practice , 2000 .

[51]  Frederick J. Gravetter,et al.  Statistics for the Behavioral Sciences [6th ed.] , 2004 .

[52]  P. Cozby,et al.  Methods in behavioral research , 1977 .

[53]  M. Banaji,et al.  Psychological. , 2015, The journals of gerontology. Series B, Psychological sciences and social sciences.

[54]  A. D. D. Groot,et al.  Perception and memory in chess: Studies in the heuristics of the prodessional eye , 1996 .

[55]  John R. Anderson,et al.  Using a Cognitive Model to Provide Instruction for a Dynamic Task , 2011, CogSci.

[56]  S. Siegel,et al.  Nonparametric Statistics for the Behavioral Sciences , 2022, The SAGE Encyclopedia of Research Design.

[57]  E. Wagenmakers,et al.  A Bayesian Perspective on Hypothesis Testing , 2006, Psychological science.

[58]  David H. Jonassen,et al.  Handbook of Individual Differences, Learning, and Instruction , 1993 .

[59]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[60]  Barry Boehm,et al.  The Spiral Model as a Tool for Evolutionary Acquisition , 2001 .

[61]  F. Ritter,et al.  How can researchers and communicators reach their desired audience on the Internet? Can Internet users be successfully and responsibly recruited as participants in studies and surveys? A recent exploration of these questions suggests some good practices. , 2001 .

[62]  Ibrahim M. Alharkan,et al.  Effects of pixel shape and color, and matrix pixel density of Arabic digital typeface on characters’ legibility , 2005 .

[63]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[64]  Joseph H. Goldberg,et al.  Identifying fixations and saccades in eye-tracking protocols , 2000, ETRA.

[65]  Mark P. Zanna,et al.  The compleat academic : a practical guide for the beginning social scientist , 1987 .

[66]  Fernand Gobet,et al.  Perception and Memory in Chess , 1996, J. Int. Comput. Games Assoc..

[67]  G. Keppel,et al.  Design and Analysis: A Researcher's Handbook , 1976 .

[68]  Jaehyon Paik.  A novel training paradigm for knowledge and skills acquisition: Hybrid schedules lead to better learning for some but not all tasks , 2011 .

[69]  Robin R. Murphy,et al.  AAAI/RoboCup-2001 Urban Search and Rescue Events , 2002, AI Mag..

[70]  W. Hays.  Methods Toward a Science of Behavior and Experience. 2nd ed. , 1985 .

[71]  Harvey S. Smallman,et al.  Naive Realism: Misplaced Faith in Realistic Displays , 2005 .

[72]  Frank E. Ritter,et al.  Developing process models as summaries of HCI action sequences , 1994 .

[73]  Andrew S. Winston,et al.  Robert Sessions Woodworth and the "Columbia Bible": How the Psychological Experiment Was Redefined , 1990 .

[74]  Michael J. Schoelles,et al.  Argus Prime: Modeling Emergent Microstrategies in a Complex, Simulated Task Environment , 1999 .

[75]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[76]  E. Donchin,et al.  The space fortress game , 1989 .

[77]  Alex Kirlik,et al.  Brunswikian Theory and Method as a Foundation for Simulation-Based Research on Clinical Judgment , 2010, Simulation in healthcare : journal of the Society for Simulation in Healthcare.

[78]  A. Everett,et al.  Orientation and Mobility Techniques: A Guide for the Practitioner , 1976 .

[79]  M. Masson.  Using confidence intervals for graphically based data interpretation. , 2003, Canadian journal of experimental psychology = Revue canadienne de psychologie experimentale.

[80]  L. Reder,et al.  What determines initial feeling of knowing? Familiarity with question terms, not with the answer , 1992 .

[81]  Allen Newell,et al.  Human Problem Solving. , 1973 .

[82]  R. Seibel.  DISCRIMINATION REACTION TIME FOR 1,023-ALTERNATIVE TASK. , 1963, Journal of experimental psychology.