A Textual Feedback Tool for Empowering Participants in Usability and UX Evaluations

ABSTRACT The usability movement has historically always sought to empower end-users of computers so that they understand what is happening and can control the outcome. In this article, we develop and evaluate a “Textual Feedback” tool for usability and user experience (UX) evaluation that can be used to empower well-educated but low-status users in UX evaluations in countries and contexts with high power distances. The proposed tool contributes to the Human–Computer Interaction (HCI) community’s pool of localized UX evaluation tools. We evaluate the tool with 40 users from two socio-economic groups in real-life UX usability evaluations settings in Malaysia. The results indicate that the Textual Feedback tool may help participants to give their thoughts in UX evaluation in high power distance contexts. In particular, the Textual Feedback tool helps high status females and low status males express more UX problems than they can with traditional concurrent think aloud (CTA) alone. We found that classic concurrent think aloud UX evaluation works fine in high power contexts, but only with the addition of Textual Feedback to mitigate the effects of socio-economic status in certain user groups. We suggest that future research on UX evaluation look more into how to empower certain user groups, such as low status female users, in UX evaluations done in high power distance contexts.

[1]  Tingting Zhao,et al.  Exploring Think-Alouds in Usability Testing: An International Survey , 2012, IEEE Transactions on Professional Communication.

[2]  Edwin H. Blake,et al.  Being participated: a community approach , 2010, PDC '10.

[3]  Mark C. Fox,et al.  Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. , 2011, Psychological bulletin.

[4]  G. Verbeke,et al.  Statistical inference in generalized linear mixed models: a review. , 2006, The British journal of mathematical and statistical psychology.

[5]  Heike Winschiers-Theophilus,et al.  Penan's Oroo' Short Message Signs (PO-SMS): Co-design of a Digital Jungle Sign Language Application , 2015, INTERACT.

[6]  Myung Hwan Yun,et al.  Usability of consumer electronic products , 2001 .

[7]  R Core Team,et al.  R: A language and environment for statistical computing. , 2014 .

[8]  Virpi Roto,et al.  UX Curve: A method for evaluating long-term user experience , 2011, Interact. Comput..

[9]  Torkil Clemmensen,et al.  Virtual Teams and Human Work Interaction Design - Learning to Work in and Designing for Virtual Teams , 2009, INTERACT.

[10]  Haralambos Mouratidis,et al.  Website Design Guidelines: High Power Distance and High-Context Culture , 2009 .

[11]  Mary Lacity,et al.  Understanding Qualitative Data: A Framework of Text Analysis Methods , 1994, J. Manag. Inf. Syst..

[12]  Alvin Yeo,et al.  Cultural effects in usability assessment , 1998, CHI Conference Summary.

[13]  Nalini Kotamraju,et al.  Playing stupid, caring for users, and putting on a good show: Feminist acts in usability study work , 2011, Interact. Comput..

[14]  Pei-Luen Patrick Rau,et al.  Cross-cultural influence on communication effectiveness and user interface design , 2009 .

[15]  Carolyn Penstein Rosé,et al.  Orality-Grounded HCID: Understanding the Oral User , 2009 .

[16]  Khalil Md Nor,et al.  Website Fit: An Overview , 2012 .

[17]  Yan Liu,et al.  Leveraging and Integrating Eastern and Western Insights for Human Engagement Studies in HCI , 2015, CHI Extended Abstracts.

[18]  E. W. Gould,et al.  Crosscurrents: cultural dimensions and global Web user-interface design , 2000, INTR.

[19]  Jennifer Ann Rode,et al.  A theoretical agenda for feminist HCI , 2011, Interact. Comput..

[20]  Vanessa Evers.  Cross-cultural applicability of user evaluation methods: a case study amongst Japanese, North-American, English and Dutch users , 2002, CHI Extended Abstracts.

[21]  Aaron Marcus,et al.  Toward an Information Society for All: HCI Challenges and R&D Recommendations , 1999, Int. J. Hum. Comput. Interact..

[22]  Abdullah Gani,et al.  Cloud Adoption in Malaysia: Trends, Opportunities, and Challenges , 2015, IEEE Cloud Computing.

[23]  Aaron Marcus,et al.  User Validation of Cultural Dimensions of a Website Design , 2007, HCI.

[24]  D. Bates,et al.  Linear Mixed-Effects Models using 'Eigen' and S4 , 2015 .

[25]  Alistair A. Young,et al.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) , 2017, MICCAI 2017.

[26]  L. Alcoff,et al.  Cultural Feminism versus Post-Structuralism: The Identity Crisis in Feminist Theory , 1988, Signs: Journal of Women in Culture and Society.

[27]  Ronald Fischer.  Fundamental Questions in Cross-Cultural Psychology: About chicken and eggs: four methods for investigating culture–behaviour links , 2011 .

[28]  K. A. Ericsson,et al.  Protocol Analysis: Verbal Reports as Data , 1984 .

[29]  R. Nisbett,et al.  Culture and systems of thought: holistic versus analytic cognition. , 2001, Psychological review.

[30]  Torkil Clemmensen,et al.  The Cultural Usability (CULTUSAB) Project: Studies of Cultural Models in Psychological Usability Evaluation Methods , 2007, HCI.

[31]  Ted Boren,et al.  Thinking aloud: reconciling theory and practice , 2000 .

[32]  Paul Dourish,et al.  Postcolonial computing: a lens on design and development , 2010, CHI.

[33]  Louise Barkhuus,et al.  From Mice to Men - 24 Years of Evaluation in CHI , 2007, CHI 2007.

[34]  Yair Amichai-Hamburger,et al.  E-empowerment: Empowerment by the Internet , 2008, Comput. Hum. Behav..

[35]  Torkil Clemmensen,et al.  Extracting User's Data: Towards development of a cultural and semantically sensitive combinatorial methodology , 2008 .

[36]  Morten Hertzum,et al.  Problem Prioritization in Usability Evaluation: From Severity Assessments Toward Impact on Design , 2006, Int. J. Hum. Comput. Interact..

[37]  Aihwa Ong.  Ecologies of Expertise: Assembling Flows, Managing Citizenship , 2008 .

[38]  Wan Fatimah Wan Ahmad,et al.  Benefits of Complementing Eye-Tracking Analysis with Think-Aloud Protocol in a Multilingual Country with High Power Distance , 2014 .

[39]  O. Ashenfelter,et al.  Estimating the Effect of Training Programs on Earnings , 1976 .

[40]  Qingxin Shi,et al.  Language Issues in Cross Cultural Usability Testing: A Pilot Study in China , 2007, HCI.

[41]  E. Krahmer,et al.  Thinking about thinking aloud: a comparison of two verbal protocols for usability testing , 2004, IEEE Transactions on Professional Communication.

[42]  Kiyoshi Kurokawa,et al.  Gender in Science , 2006 .

[43]  Y. Connie Yuan,et al.  Expertise Recognition and Influence in Intercultural Groups: Differences Between Face-to-Face and Computer-Mediated Communication , 2013, J. Comput. Mediat. Commun..

[44]  M. Cortazzi,et al.  Changing Practices in Chinese Cultures of Learning , 2006 .

[45]  Torkil Clemmensen,et al.  Towards a Theory of Cultural Usability: A Comparison of ADA and CM-U Theory , 2009, HCI.

[46]  Qingxin Shi,et al.  A field study of the relationship and communication between Chinese evaluators and users in thinking aloud usability tests , 2008, NordiCHI.

[47]  Joey F. George,et al.  Media Selection as a Strategic Component of Communication , 2013, MIS Q..

[48]  E. Mulvey,et al.  Regression analyses of counts and rates: Poisson, overdispersed Poisson, and negative binomial models. , 1995, Psychological bulletin.

[49]  T. Devos,et al.  Silent or Talking in the Classroom: Implicit Self-Stereotyping Among Asian and White Students , 2014 .

[50]  Daniel Rosenberg,et al.  Leading global UX teams , 2011, INTR.

[51]  James R. Lewis.  Introduction to the Special Issue on Usability and User Experience: Methodological Evolution , 2015, Int. J. Hum. Comput. Interact..

[52]  Torkil Clemmensen,et al.  Website Usability in Asia “From Within”: An Overview of a Decade of Literature , 2013, Int. J. Hum. Comput. Interact..

[53]  Sara B. Kiesler,et al.  The Equalization Phenomenon: Status Effects in Computer-Mediated and Face-to-Face Decision-Making Groups , 1991, Hum. Comput. Interact..

[54]  Tingting Zhao,et al.  Keep talking: an analysis of participant utterances gathered using two concurrent think-aloud methods , 2010, NordiCHI.

[55]  José L. Abdelnour-Nocera,et al.  Harambee: a novel usability evaluation method for low-end users in Kenya , 2014, NordiCHI.

[56]  Morten Hertzum,et al.  Cultural cognition in usability evaluation , 2009, Interact. Comput..

[57]  Heejung S. Kim,et al.  We talk, therefore we think? A cultural analysis of the effect of talking on thinking. , 2002, Journal of personality and social psychology.

[58]  Andy Smith,et al.  Issues in Adapting Usability Testing for Global Usability , 2011, Global Usability.

[59]  E. W. Gould,et al.  Applying culture to Web site design: a comparison of Malaysian and US Web sites , 2000, 18th Annual Conference on Computer Documentation. ipcc sigdoc 2000. Technology and Teamwork. Proceedings. IEEE Professional Communication Society International Professional Communication Conference an.

[60]  Andy Smith,et al.  Evaluation methods and cultural differences: studies across three continents , 2008, NordiCHI.

[61]  R. Pineda.  Technology in culture : a theoretical discourse on convergence in human-technology interaction , 2014 .

[62]  Kim Nee Goh,et al.  A Comparison of Usability Testing Methods for an E-Commerce Website: A Case Study on a Malaysia Online Gift Shop , 2013, 2013 10th International Conference on Information Technology: New Generations.

[63]  James R. Lewis,et al.  Usability: Lessons Learned … and Yet to Be Learned , 2014, Int. J. Hum. Comput. Interact..

[64]  Heike Winschiers-Theophilus,et al.  Assumptions Considered Harmful , 2007, HCI.

[65]  Yanan Chen,et al.  Do Usability Professionals Think about User Experience in the Same Way as Users and Developers Do? , 2013, INTERACT.

[66]  Anirudha Joshi,et al.  Institutionalizing HCI in Asia , 2007, INTERACT.

[67]  Bilge Mutlu,et al.  How social distance shapes human-robot interaction , 2014, Int. J. Hum. Comput. Stud..

[68]  Alvin W. Yeo,et al.  Are Usability Assessment Techniques Reliable in Non‐Western Cultures? , 2000, Electron. J. Inf. Syst. Dev. Ctries..

[69]  Zanariah Jano,et al.  Website Usability and Cultural Dimensions in Malaysian and Australian Universities , 2015 .