Personas and Analytics: A Comparative User Study of Efficiency and Effectiveness for a User Identification Task

Personas are a well-known technique in human computer interaction. However, there is a lack of rigorous empirical research evaluating personas relative to other methods. In this 34-participant experiment, we compare a persona system and an analytics system, both using identical user data, for efficiency and effectiveness for a user identification task. Results show that personas afford faster task completion than the analytics system, as well as outperforming analytics with significantly higher user identification accuracy. Qualitative analysis of think-aloud transcripts shows that personas have other benefits regarding learnability and consistency. However, the analytics system affords insights and capabilities that personas cannot due to inherent design differences. Findings support the use of personas to learn about users, empirically confirming some of the stated benefits in the literature, while also highlighting the limitations of personas that may necessitate the use of accompanying methods.

[1]  NielsenLene,et al.  A Template for Design Personas , 2015 .

[2]  Joseph H. Goldberg,et al.  Eye tracking in web search tasks: design implications , 2002, ETRA.

[3]  Chris Chapman,et al.  The Personas' New Clothes: Methodological and Practical Arguments against a Popular Method , 2006 .

[4]  Morteza Dehghani,et al.  Conversation level syntax similarity metric , 2017, Behavior Research Methods.

[5]  Jianming Dong,et al.  Getting the Most Out of Personas for Product Usability Enhancements , 2007, HCI.

[6]  Alan Cooper,et al.  The Inmates Are Running the Asylum: Why High Tech Products Drive Us Crazy and How to Restore the Sanity (2nd Edition) , 1999 .

[7]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[8]  Jan Stage,et al.  A Template for Design Personas: Analysis of 47 Persona Descriptions from Danish Industries and Organizations , 2015, Int. J. Sociotechnology Knowl. Dev..

[9]  Susanne Bødker,et al.  Personas, people and participation: challenges from the trenches of local government , 2012, PDC '12.

[10]  Yasuhiro Watanabe,et al.  ID3P: Iterative Data-Driven Development of Persona Based on Quantitative Evaluation and Revision , 2017, 2017 IEEE/ACM 10th International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE).

[11]  John S. Pruitt,et al.  The Persona Lifecycle: Keeping People in Mind Throughout Product Design , 2006 .

[12]  Chris Chapman,et al.  Quantitative Evaluation of Personas as Information , 2008 .

[13]  Xiang Zhang,et al.  Data-driven Personas: Constructing Archetypal Users with Clickstreams and User Telemetry , 2016, CHI.

[14]  Benjamin Kuipers,et al.  A Description of Think Aloud Method and Protocol Analysis , 1993 .

[15]  Adrienne Massanari,et al.  Designing for imaginary friends: information architecture, personas and the politics of user-centered design , 2010, New Media Soc..

[16]  Elina Eriksson,et al.  The secret life of a persona: when the personal becomes private , 2013, CHI.

[17]  Tamara Sumner,et al.  A latent semantic analysis methodology for the identification and creation of personas , 2008, CHI.

[18]  Panagiotis Zaharias,et al.  Minimizing Attrition in Online Degree Courses. , 2009 .

[19]  Bernard J. Jansen,et al.  Detecting Demographic Bias in Automatically Generated Personas , 2019, CHI Extended Abstracts.

[20]  Rosa Gudjonsdottir,et al.  Personas and scenarios: design tool or a communication device? , 2008, COOP.

[21]  Liyana Shuib,et al.  Big data adoption: State of the art and research challenges , 2019, Inf. Process. Manag..

[22]  Bernard J. Jansen,et al.  Customer segmentation using online platforms: isolating behavioral and demographic segments for persona creation via aggregated user data , 2018, Social Network Analysis and Mining.

[23]  Steven Walczak,et al.  Comparing Semi-Automated Clustering Methods for Persona Development , 2012, IEEE Transactions on Software Engineering.

[24]  Heikki Karjaluoto,et al.  The use of Web analytics for digital marketing performance measurement , 2015 .

[25]  John M. Carroll,et al.  Creating Persona Skeletons from Imbalanced Datasets - A Case Study using U.S. Older Adults' Health Data , 2019, Conference on Designing Interactive Systems.

[26]  Margaret M. Burnett,et al.  Gender-Inclusiveness Personas vs. Stereotyping: Can We Have it Both Ways? , 2017, CHI.

[27]  Bernard J. Jansen,et al.  “Is More Better?”: Impact of Multiple Photos on Perception of Persona Profiles , 2018, CHI.

[28]  Eric J. Johnson,et al.  The validity of verbal protocols , 1989, Memory & cognition.

[29]  Tomasz Miaskiewicz,et al.  Personas and user-centered design: How can personas benefit product design processes? , 2011 .

[30]  Ayoub Ait Lahcen,et al.  Towards a real-time processing framework based on improved distributed recurrent neural network variants with fastText for social big data analytics , 2020, Inf. Process. Manag..

[31]  Bernard J. Jansen,et al.  Imaginary People Representing Real Numbers , 2018, ACM Trans. Web.

[32]  Bernard J. Jansen,et al.  Conversion potential: a metric for evaluating search engine advertising performance , 2017 .

[33]  Nigel Bevan,et al.  Human-Computer Interaction Standards , 1995 .

[34]  Kim Goodwin,et al.  Designing for the Digital Age: How to Create Human-Centered Products and Services , 2009 .

[35]  Michael F. Clarke,et al.  The Work of Mad Men that Makes the Methods of Math Men Work: Practically Occasioned Segment Design , 2015, International Conference on Human Factors in Computing Systems.

[36]  Karen Holtzblatt,et al.  Contextual design , 1997, INTR.

[37]  Gary James Jason,et al.  The Logic of Scientific Discovery , 1988 .

[38]  Yvonne Dittrich,et al.  Personas is not applicable: local remedies interpreted in a wider context , 2004, PDC 04.

[39]  Kari Rönkkö,et al.  An Empirical Study Demonstrating How Different Design Constraints, Project Organization and Contexts Limited the Utility of Personas , 2005, Proceedings of the 38th Annual Hawaii International Conference on System Sciences.

[40]  Ann Blandford,et al.  The challenges of delivering validated personas for medical equipment design. , 2014, Applied ergonomics.

[41]  Lene Nielsen,et al.  Personas - User Focused Design , 2012, Human–Computer Interaction Series.

[42]  Bernard J. Jansen,et al.  Personas Changing Over Time: Analyzing Variations of Data-Driven Personas During a Two-Year Period , 2019, CHI Extended Abstracts.

[43]  Jang Hyun Kim,et al.  Instagram user characteristics and the color of their photos: Colorfulness, color diversity, and color harmony , 2019, Inf. Process. Manag..

[44]  Frank Y. Guo,et al.  Creating Effective Personas for Product Design: Insights from a Case Study , 2011, HCI.

[45]  Jonathan Grudin,et al.  Personas: practice and theory , 2003, DUX '03.

[46]  David A. Hull.  Using statistical testing in the evaluation of retrieval experiments , 1993, SIGIR.

[47]  Bernard J. Jansen,et al.  Are personas done? Evaluating their usefulness in the age of digital analytics , 2018, Persona Studies.

[48]  Karen Holtzblatt,et al.  Rapid Contextual Design: A How-To Guide to Key Techniques for User-Centered Design , 2004, UBIQ.

[49]  Tomasz Miaskiewicz,et al.  A Preliminary Examination of Using Personas to Enhance User-Centered Design , 2009, AMCIS.

[50]  A. Bandura.  Self-efficacy mechanism in human agency , 2024, Psihologìâ ì suspìlʹstvo.

[51]  Lene Nielsen,et al.  Developing international personas: A new intercultural communication practice in globalized societies , 2017 .

[52]  Steve Whittaker,et al.  How do designers and user experience professionals actually perceive and use personas? , 2012, CHI.

[53]  Jonathan Grudin,et al.  Personas, Participatory Design and Product Development: An Infrastructure for Engagement , 2002 .

[54]  HoltzblattKaren,et al.  Rapid Contextual Design , 2005 .

[55]  Bernard J. Jansen,et al.  Persona Generation from Aggregated Social Media Data , 2017, CHI Extended Abstracts.

[56]  Christopher A. Mattson,et al.  The Personification of Big Data , 2019, Proceedings of the Design Society: International Conference on Engineering Design.

[57]  Panagiotis Zaharias,et al.  Usability in the Context of e-Learning: A Framework Augmenting 'Traditional' Usability Constructs with Instructional Design and Motivation to Learn , 2009, Int. J. Technol. Hum. Interact..

[58]  Cindy LeRouge,et al.  Introducing User Profiles and Personas into Information Systems Development , 2007, AMCIS.

[59]  José L. Abdelnour-Nocera,et al.  A Critique of Personas as representations of "the other" in Cross-Cultural Technology Design , 2016, AfriCHI.

[60]  Iftikhar Ahmed,et al.  Making personas memorable , 2007, CHI Extended Abstracts.

[61]  Anand K. Gramopadhye,et al.  A Case Study on Use of Personas in Design and Development of an Audit Management System , 2007 .

[62]  Thomas Ertl,et al.  Triangulating user behavior using eye movement, interaction, and think aloud data , 2016, ETRA.

[63]  Erin Friess,et al.  Personas and decision making in the design process: an ethnographic case study , 2012, CHI.

[64]  Wendell R. Smith.  Product Differentiation and Market Segmentation as Alternative Marketing Strategies , 1956 .

[65]  Pam J. Mayhew,et al.  To Intervene or Not to Intervene:An Investigation of Three Think-Aloud Protocols in Usability Testing , 2017 .