The Impact of Usability Reports and User Test Observations on Developers' Understanding of Usability Data: An Exploratory Study

A usability evaluation provides a strong and rich basis for understanding and improving the design of user interaction with a software system. Exploiting this evaluation requires feedback that significantly impacts the developers' understanding of usability data about the interaction design of the system. This article presents results from an exploratory study of 2 ways of providing feedback from a usability evaluation: observation of user tests and reading usability reports. A case study and a field experiment were used to explore how observation and usability reports impact developers' understanding of usability data. The results indicate that observation of user tests facilitated a rich understanding of usability problems and created empathy with the users and their work. The usability report had a strong impact on the developers' understanding of specific usability problems and supported a systematic approach to deal effectively with problems.

[1]  Klaus Kaasgaard,et al.  Comparative usability evaluation , 2004, Behav. Inf. Technol..

[2]  Debora Shaw,et al.  Handbook of usability testing: How to plan, design, and conduct effective tests , 1996 .

[3]  Kasper Hornbæk,et al.  Comparing usability problems and redesign proposals as input to practical systems development , 2005, CHI.

[4]  Chris Fowler,et al.  Using the usability laboratory: BT's experiences , 1994, Behav. Inf. Technol..

[5]  Mary Czerwinski,et al.  Organizing usability work to fit the full product range , 1999, CACM.

[6]  Kasper Hornbæk,et al.  Input from usability evaluation in the form of problems and redesigns : results from interviews with developers , 2004 .

[7]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[8]  Kim Halskov,et al.  Cooperative usability practices , 1999, CACM.

[9]  Rick Spencer,et al.  The streamlined cognitive walkthrough method, working around social constraints encountered in a software development company , 2000, CHI.

[10]  Marilyn C. Salzman,et al.  Smoke and mirrors: setting the stage for a successful usability test , 1994, Behav. Inf. Technol..

[11]  Donald A. Norman,et al.  The invisible computer , 1998 .

[12]  Steve Miller,et al.  Designing and equipping a usability laboratory , 1994, Behav. Inf. Technol..

[13]  Klaus B. Bærentsen,et al.  A contribution to the design process , 1999, CACM.

[14]  Joseph S. Dumas,et al.  Usability in practice: formative usability evaluations - evolution and revolution , 2002, CHI Extended Abstracts.

[15]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[16]  Manuel A. Pérez-Quiñones,et al.  Usability inspection of digital libraries: a case study , 2004, International Journal on Digital Libraries.

[17]  Donald O. Case,et al.  The Trouble with Computers: Usefulness, Usability, and Productivity, by Thomas K. Landauer , 1996, J. Am. Soc. Inf. Sci..

[18]  Arnold M. Lund,et al.  Ameritech's usability laboratory: from prototype to final design , 1994, Behav. Inf. Technol..

[19]  Desirée Sy,et al.  Bridging the communication gap in the workplace with usability engineering , 1994, SIGDOC '94.

[20]  Lynne Baillie.  Future Telecommunications: Exploring Actual Use , 2003, INTERACT.

[21]  Steven Pace,et al.  A grounded theory of the flow experiences of Web users , 2004, Int. J. Hum. Comput. Stud..

[22]  Thomas K. Landauer,et al.  Trouble with Computers: Usefulness, Usability, and Productivity , 1996 .

[23]  Kate Ehrlich,et al.  Getting the whole team into usability testing , 1994, IEEE Software.

[24]  Lars Mathiassen,et al.  Reflective Systems Development , 1998, Scand. J. Inf. Syst..

[25]  William R. Dolan,et al.  A flexible approach to third-party usability , 1999, CACM.

[26]  Deborah J. Mayhew,et al.  The usability engineering lifecycle , 1998, CHI Conference Summary.

[27]  Jakob Nielsen,et al.  Chapter 4 – The Usability Engineering Lifecycle , 1993 .

[28]  Dieter Zirkler,et al.  Usability testing in a competive market: lessons learned , 1994, Behav. Inf. Technol..

[29]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[30]  Gilbert Cockton,et al.  From quality in use to value in the world , 2004, CHI EA '04.

[31]  Janet L. Fath,et al.  A practical guide to using software usability labs: lessons learned at IBM , 1994, Behav. Inf. Technol..

[32]  Izak Benbasat,et al.  The Case Research Strategy in Studies of Information Systems , 1987, MIS Q..

[33]  Janice Anne Rohn.  The usability engineering laboratories at Sun Microsystems , 1994, Behav. Inf. Technol..

[34]  Thomas P. Moran.  Everyday adaptive design , 2002, DIS '02.

[35]  Jan Stage,et al.  Instant data analysis: conducting usability evaluations in a day , 2004, NordiCHI '04.

[36]  Deborah J. Mayhew.  Business: Strategic development of the usability engineering function , 1999, INTR.

[37]  P. Kidwell,et al.  The trouble with computers: Usefulness, usability and productivity , 1996, IEEE Annals of the History of Computing.

[38]  Steve Howard,et al.  Location, Location, Location: Challenges of Outsourced Usability Evaluation , 2004 .

[39]  Scott Lewis,et al.  Breaking away from the conventional 'usability lab': the Customer-Centered Design Group at Tektronix, Inc , 1994, Behav. Inf. Technol..

[40]  Jean Scholtz,et al.  A new usability standard and what it means to you , 2002, CHIB.