Internet-based research: providing a foundation for web-design guidelines

In this article, we propose that remote, internet-based studies of real users interacting with real websites on their own computers at a time and place convenient for them will provide a solid empirical base from which researchers can extrapolate reliable and valid web-design guidelines. After a discussion of research methods that have been used to support the principles that underlie web-design guidelines, we review some of the methodological issues associated with internet-based research and tools for supporting such work. Given advances in technology, the multitude of users online, and emerging technologies with new interfaces, the time has come for technical communication researchers to enter the arena of internet-based research and conduct remote experiments to support the web-design guidelines that they espouse.

[1]  Ulf-Dietrich Reips,et al.  A Brief History of Web Experimenting , 2000 .

[2]  Jan H. Spyridakis,et al.  Writing for Human Performance: Relating Reading Research to Document Design. , 1992 .

[3]  R. Fairlie The alertbox: Current issues in web usability , 2000 .

[4]  M. Birnbaum Human research and data collection via the internet. , 2004, Annual review of psychology.

[5]  Rune Pettersson,et al.  Information Design: An Introduction , 2002 .

[6]  Penelope M. Sanderson,et al.  Verbal Protocols as a Research Tool in Human Factors: Panel Discussion , 1990 .

[7]  R. Todd Stephens,et al.  A framework for the identification of electronic commerce design elements that enable trust within the small hotel industry , 2004, ACM-SE 42.

[8]  Jan H. Spyridakis,et al.  The relevance of reliability and validity to usability testing , 1989 .

[9]  Barbara S. Chaparro,et al.  The Impact of Paging vs. Scrolling on Reading Online Text Passages , 2003 .

[10]  M. Traugott,et al.  Web survey design and administration. , 2001, Public opinion quarterly.

[11]  Marti A. Hearst,et al.  Empirically validated web page design metrics , 2001, CHI.

[12]  K. McGraw,et al.  The Integrity of Web-Delivered Experiments: Can You Trust the Data? , 2000, Psychological science.

[13]  Barry C. Gribbons,et al.  True and Quasi-Experimental Designs , 1996 .

[14]  Erin E. Michalak,et al.  Report European Psychologist V Guidelines for Internet Research : An Update , 2004 .

[15]  Petra Schubert,et al.  Web assessment-a model for the evaluation and the assessment of successful electronic commerce applications , 1998, Proceedings of the Thirty-First Hawaii International Conference on System Sciences.

[16]  J. Chiou,et al.  Should a company have message boards on its Web sites? , 2003 .

[17]  Attila Szabo,et al.  Guidelines for Internet Research , 1998 .

[18]  Thomas R. Williams Guidelines for Designing and Evaluating the Display of Information on the Web , 2000 .

[19]  D. Bachiochi,et al.  Usability Studies And Sesigning Navigational Aids for the World Wide Web , 1997, Comput. Networks.

[20]  Peter Yule,et al.  Express: A Web-based technology to support human and computational experimentation , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[21]  Barbara S. Chaparro,et al.  Examining the Effects of Hypertext Shape on User Performance , 2002 .

[22]  M. Birnbaum Decision Making in the Lab and on the Web , 2000 .

[23]  Hock-Hai Teo,et al.  An empirical study of the effects of interactivity on web user attitude , 2003 .

[24]  Lenita M. Davis,et al.  Empirical testing of a model of online store atmospherics and shopper responses , 2003 .

[25]  Jared M. Spool,et al.  Testing web sites: five users is nowhere near enough , 2001, CHI Extended Abstracts.

[26]  S. Gosling,et al.  Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. , 2004, The American psychologist.

[27]  Youngkee Kim How Do People Evaluate a Web Site's Credibility , 2007 .

[28]  Byeong-Min Yu,et al.  The effects of menu design on information-seeking performance and user's attitude on the World Wide Web , 2002, J. Assoc. Inf. Sci. Technol..

[29]  Dianna L. Newman,et al.  The relationship of student interest and advance organizer effectiveness , 1988 .

[30]  Jeffrey Heer,et al.  WebQuilt : A Proxy-based Approach to Remote Web Usability Testing , 2001 .

[31]  M. Banaji,et al.  Psychological. , 2015, The journals of gerontology. Series B, Psychological sciences and social sciences.

[32]  Michael H. Birnbaum,et al.  Introduction to Behavioral Research on the Internet , 2000 .

[33]  Morten Hertzum,et al.  The Evaluator Effect in Usability Studies: Problem Detection and Severity Judgments , 1998 .

[34]  Charles F. Hofacker,et al.  Using server log files and online experiments to enhance internet marketing , 2005 .

[35]  Jakob Nielsen,et al.  Homepage Usability: 50 Websites Deconstructed , 2001 .

[36]  Fabio Paternò,et al.  Tools for remote usability evaluation of Web applications through browser logs and task models , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[37]  Robert J. Tierney,et al.  Reading Strategies and Practices: A Compendium , 1985 .

[38]  James Hartley,et al.  Designing instructional text for older readers: a literature review , 1994, Br. J. Educ. Technol..

[39]  Varun Grover,et al.  Investigating the Impact of Web Site Value and Advertising on Firm Performance in Electronic Commerce , 2002, Int. J. Electron. Commer..

[40]  Chang Liu,et al.  Exploring the factors associated with Web site success in the context of electronic commerce , 2000, Inf. Manag..

[41]  G. Johnson Internet Research Methods: A Practical Guide for the Social and Behavioural Sciences , 2003 .

[42]  Ann Marie Fiore,et al.  Influence of image interactivity on approach responses towards an online retailer , 2003, Internet Res..

[43]  Alastair G. Smith Homepage Usability: 50 Web sites Deconstructed , 2003 .

[44]  Miriam J. Metzger,et al.  The perceived credibility of personal Web page information as influenced by the sex of the source , 2003, Comput. Hum. Behav..

[45]  Christian Köhler,et al.  How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews , 2002, BMJ : British Medical Journal.

[46]  R. Rosenthal Experimenter effects in behavioral research , 1968 .

[47]  Ben Shneiderman,et al.  Performance Benefits of Simultaneous Over Sequential Menus as Task Complexity Increases , 2000, Int. J. Hum. Comput. Interact..

[48]  R. Rosenthal,et al.  Artifact in behavioral research , 1969 .

[49]  Marti A. Hearst,et al.  Statistical profiles of highly-rated web sites , 2002, CHI.

[50]  O. Byung Kwon,et al.  Impact of website information design factors on consumer ratings of web-based auction sites , 2002, Behav. Inf. Technol..

[51]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[52]  Ulf-Dietrich Reips Standards for Internet-based experimenting. , 2002, Experimental psychology.

[53]  Jakob Nielsen,et al.  Applying writing guidelines to Web pages , 1998, CHI Conference Summary.

[54]  Stefan Stieger,et al.  Scientific LogAnalyzer: A Web-based tool for analyses of server log files in psychological research , 2004, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[55]  Jeffrey Heer,et al.  What did they do? understanding clickstreams with the WebQuilt visualization system , 2002, AVI '02.

[56]  Jan H. Spyridakis,et al.  Influencing web-browsing behavior with intriguing and informative hyperlink wording , 2005, J. Inf. Sci..

[57]  Carolyn Snyder,et al.  Web Site Usability: A Designer's Guide , 1997 .

[58]  Daniel Cunliffe,et al.  Developing usable Web sites - a review and model , 2000, Internet Res..

[59]  Anand Kumar,et al.  Web Commercials and Advertising Hierarchy-of-Effects , 2000, Journal of Advertising Research.

[60]  J. Horrigan,et al.  Trust and privacy online: Why Americans want to rewrite the rules , 2000 .

[61]  Peter Jan Schellens,et al.  Toward a document evaluation methodology: what does research tell us about the validity and reliability of evaluation methods? , 2000 .

[62]  Steven D. Penrod,et al.  Methodological variables in Web-based research that may affect results: Sample type, monetary incentives, and personal information , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[63]  Jamie Murphy Surfers and Searchers , 1999 .

[64]  Lee Rainie,et al.  Getting Serious Online. , 2002 .

[65]  Karen A. Schriver Dynamics in document design , 1998 .

[66]  Dieter Fink,et al.  Perceptions of Web site design characteristics: a Malaysian/Australian comparison , 2000, Internet Res..

[67]  Andy Edmonds,et al.  Uzilla: A new tool for Web usability testing , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[68]  Lars Schmidt,et al.  Comparative evaluation of usability tests , 1999, CHI Extended Abstracts.

[69]  Jakob Nielsen,et al.  Concise, SCANNABLE, and Objective: How to Write for the Web , 2006 .

[70]  Thomas A. Powell Web Design: The Complete Reference , 2000 .

[71]  Jinwoo Kim,et al.  Contextual Navigation Aids for Two World Wide Web Systems , 2000, Int. J. Hum. Comput. Interact..

[72]  R. Rosenthal,et al.  Experimenter effects in behavioral research , 1968 .

[73]  Jan H. Spyridakis,et al.  The Influence of Semantics and Syntax on what Readers Remember , 1999 .

[74]  M H Birnbaum,et al.  SurveyWiz and factorWiz: Javascript Web pages that make HTML forms for research on the internet , 2000, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[75]  Melody Y. Ivory,et al.  Automated web site evaluation - researchers and practitioners perspectives , 2010, Human-computer interaction series.

[76]  Dyi-Yih Michael Lin,et al.  Evaluating older adults' retention in hypertext perusal: impacts of presentation media as a function of text topology , 2004, Comput. Hum. Behav..

[77]  Ming-Hui Huang,et al.  Designing website attributes to induce experiential encounters , 2003, Comput. Hum. Behav..

[78]  Pei-Luen Patrick Rau,et al.  Culture differences in design of portal sites , 2003, Ergonomics.

[79]  Michael Marsiske,et al.  Text processing variables predict the readability of everyday documents read by older adults. , 1993, Reading research quarterly.

[80]  Ellen R. Tauber,et al.  Experts vs. Online Consumers: A Comparative Credibility Study of Health and Finance Web Sites , 2002 .

[81]  Sarah Waterson,et al.  In the lab and out in the wild: remote web usability testing for mobile devices , 2002, CHI Extended Abstracts.

[82]  Titus Schleyer,et al.  Research Paper: Methods for the Design and Administration of Web-based Surveys , 2000, J. Am. Medical Informatics Assoc..

[83]  Nikunj P. Dalal,et al.  Cognitive design of home pages: an experimental study of comprehension on the World Wide Web , 2000, Inf. Process. Manag..

[84]  Paul van Schaik,et al.  Using on-line surveys to measure three key constructs of the quality of human-computer interaction in web sites: psychometric properties and implications , 2003, Int. J. Hum. Comput. Stud..

[85]  J. Ramey Guidelines for Web data collection , 2000 .

[86]  Gordon C. Bruner,et al.  Webpage Background and Viewer Attitudes , 2000, Journal of Advertising Research.

[87]  Robert Mittman,et al.  Health e-People: The Online Consumer Experience , 2000 .

[88]  Jan H. Spyridakis,et al.  The effect of heading frequency on comprehension of online information , 2002, Proceedings. IEEE International Professional Communication Conference.

[89]  J.H. Spyridakis,et al.  A tool for supporting Web-based empirical research: providing a basis for Web design guidelines , 2004, International Professional Communication Conference, 2004. IPCC 2004. Proceedings..

[90]  Marti A. Hearst,et al.  Improving Web Site Design , 2002, IEEE Internet Comput..

[91]  Jonathan W. Palmer,et al.  Web Site Usability, Design, and Performance Metrics , 2002, Inf. Syst. Res..

[92]  Michaël Steehouder,et al.  Role Playing on the Web: Guidelines for Designing and Evaluating Personas Online , 2000 .

[93]  Vitaveska Lanfranchi,et al.  Remote Web usability testing , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[94]  Walter Kintsch,et al.  Comprehension: A Paradigm for Cognition , 1998 .

[95]  Jan H. Spyridakis,et al.  The effect of hyperlink wording on user performance , 2002 .

[96]  Brian H. Bornstein,et al.  Web-based research: Methodological variables’ effects on dropout and sample characteristics , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[97]  Mark D. Tew,et al.  PsychExps: An Online Psychology Laboratory , 2000 .

[98]  Jan H. Spyridakis,et al.  Hypertext: a new tool and its effect on audience comprehension , 1991, IPCC 91 Proceedings The Engineered Communication.

[99]  Barbara S. Chaparro,et al.  Cascading versus Indexed Menu Design , 2003 .

[100]  X Drèze,et al.  A web-based methodology for product design evaluation and optimisation , 1998, J. Oper. Res. Soc..

[101]  Jan H. Spyridakis,et al.  Guidelines for Authoring Comprehensible Web Pages and Evaluating Their Success , 2000 .

[102]  Ulf-Dietrich Reips Chapter 4 – The Web Experiment Method: Advantages, Disadvantages, and Solutions , 2000 .

[103]  Reeshad S. Dalal,et al.  Validity of Web-Based Psychological Research , 2000 .

[104]  Michael A. Hughes Rigor in Usability Testing. , 1999 .

[105]  Ajit Kambil,et al.  Consumer Behavior in Web-Based Commerce: An Empirical Study , 2001, Int. J. Electron. Commer..

[106]  Mark Sadoski,et al.  A Causal Model of Sentence Recall: Effects of Familiarity, Concreteness, Comprehensibility, and Interestingness , 1993 .

[107]  Anthony G. Greenwald,et al.  E‐Research: Ethics, Security, Design, and Control in Psychological Research on the Internet , 2002 .

[108]  K. Freeman,et al.  An Examination of Factors That Affect the Credibility of Online Health Information , 2004 .

[109]  Gerhard Deffner,et al.  Verbal Protocols as a Research Tool in Human Factors: Symposium Abstract , 1990 .

[110]  Connie K. Varnhagen,et al.  Effects of Surrounding Information and Line Length on Text Comprehension from the Web , 2002 .

[111]  Rashmi Sinha Content or Graphics? An Empirical Analysis of Criteria for Award-Winning Websites , 2001 .

[112]  Barbara S. Chaparro,et al.  Top Ten Mistakes of Shopping Cart Design Revisited: A Survey of 500 Top E-Commerce Websites , 2007 .

[113]  David K. Farkas,et al.  Guidelines for Designing Web Navigation , 2000 .

[114]  Jan H. Spyridakis,et al.  A Web-based study of user performance with enhanced local navigational cues , 2002, Proceedings. IEEE International Professional Communication Conference.

[115]  L. Faulkner Beyond the five-user assumption: Benefits of increased sample sizes in usability testing , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[116]  Guy W. Mullarkey,et al.  Factors Affecting Online Advertising Recall: A Study of Students , 2003, Journal of Advertising Research.

[117]  Ulf-Dietrich Reips,et al.  WEXTOR: A Web-based tool for generating and visualizing experimental designs and procedures , 2002, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[118]  B. Tuckman Conducting Educational Research , 1972 .

[119]  Barbara S. Chaparro,et al.  Can Expanding Targets Make Object Selection Easier for Older Adults , 2003 .