Analysis in practical usability evaluation: a survey study

Analysis is a key part of conducting usability evaluations, yet rarely systematically studied. Thus, we lack direction on how to do research on supporting practitioners' analysis and lose an opportunity for practitioners to learn from each other. We have surveyed 155 usability practitioners on the analysis in their latest usability evaluation. Analysis is typically flexible and light-weight. At the same time, practitioners see a need to strengthen reliability in evaluation. Redesign is closely integrated with analysis; more than half of the respondents provide visual redesign suggestions in their evaluation deliverables. Analysis support from academic research, including tools, forms and structured formats, does not seem to have direct impact on analysis practice. We provide six recommendations for future research to better support analysis.

[1]  Karel Vredenburg,et al.  A survey of user-centered design practice , 2002, CHI.

[2]  Jan Gulliksen,et al.  Usability professionals - current practices and future development , 2006, Interact. Comput..

[3]  Tonya L. Smith-Jackson,et al.  Supporting novice usability practitioners with usability engineering tools , 2009, Int. J. Hum. Comput. Stud..

[4]  Malcolm P. Atkinson,et al.  Comparison of evaluation methods using structured usability problem reports , 1997, Behav. Inf. Technol..

[5]  Kasper Hornbæk,et al.  Usability Evaluation as Idea Generation , 2008, Maturing Usability.

[6]  Dennis R. Wixon,et al.  Using the RITE method to improve products; a definition and a case study , 2007 .

[7]  Ebba Þóra Hvannberg,et al.  Heuristic evaluation: Comparing ways of finding and reporting usability problems , 2007, Interact. Comput..

[8]  Cathleen Wharton,et al.  The cognitive walkthrough method: a practitioner's guide , 1994 .

[9]  Kasper Hornbæk,et al.  Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes , 2011, Int. J. Hum. Comput. Interact..

[10]  Jan Stage,et al.  Supporting problem identification in usability evaluations , 2005, OZCHI.

[11]  Kasper Hornbæk,et al.  Comparing usability problems and redesign proposals as input to practical systems development , 2005, CHI.

[12]  Stephanie Rosenbaum,et al.  The Future of Usability Evaluation: Increasing Impact on Value , 2008, Maturing Usability.

[13]  Jan Stage,et al.  The Interplay Between Usability Evaluation and User Interaction Design , 2006, Int. J. Hum. Comput. Interact..

[14]  Terence S. Andre,et al.  Determining the Effectiveness of the Usability Problem Inspector: A Theory-Based Model and Tool for Finding Usability Problems , 2003, Hum. Factors.

[15]  Ted Boren,et al.  Thinking aloud: reconciling theory and practice , 2000 .

[16]  Dana Chisnell,et al.  Handbook of Usability Testing , 2009 .

[17]  Kasper Hornbæk,et al.  What do usability evaluators do in practice?: an explorative study of think-aloud testing , 2006, DIS '06.

[18]  Jesper Kjeldskov,et al.  Aalborg Universitet Instant Data Analysis : Evaluating Usability in a Day , 2004 .

[19]  Gilbert Cockton,et al.  A Framework for Usability Problem Extraction , 1999, INTERACT.

[20]  Jan Stage,et al.  Instant data analysis: conducting usability evaluations in a day , 2004, NordiCHI '04.

[21]  Abigail Sellen,et al.  User sketches: a quick, inexpensive, and effective way to elicit more reflective user feedback , 2006, NordiCHI '06.

[22]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[23]  Kasper Hornbæk,et al.  Analysis in usability evaluations: an exploratory study , 2010, NordiCHI.

[24]  D. Ezzy.  Qualitative Analysis: Practice and Innovation , 2002 .

[25]  Marc Hassenzahl,et al.  User experience - a research agenda , 2006, Behav. Inf. Technol..

[26]  Kasper Hornbæk,et al.  Work-domain knowledge in usability evaluation: Experiences with Cooperative Usability Testing , 2010, J. Syst. Softw..

[27]  Dominic Furniss.  Beyond problem identification : valuing methods in a 'system usability practice' , 2008 .

[28]  Miranda Galadriel Capra,et al.  Usability Problem Description and the Evaluator Effect in Usability Testing , 2006 .

[29]  Gilbert Cockton,et al.  Inspection-based evaluations , 2002 .

[30]  Gilbert Cockton,et al.  Reconditioned merchandise: extended structured report formats in usability inspection , 2004, CHI EA '04.

[31]  Morten Hertzum,et al.  The Evaluator Effect: A Chilling Fact About Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[32]  Jan Gulliksen,et al.  Use and Usefulness of HCI Methods: Results from an Exploratory Study among Nordic HCI Practitioners , 2005, BCS HCI.

[33]  Susan M. Dray,et al.  How to bring HCI research and practice closer together , 2010, CHI Extended Abstracts.

[34]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[35]  Steven M. Belz,et al.  The user action framework: a reliable foundation for usability engineering support tools , 2001, Int. J. Hum. Comput. Stud..