Eye tracking scanpath analysis on web pages: how many users?

The number of users required for usability studies has been a controversial issue over 30 years. Some researchers suggest a certain number of users to be included in these studies. However, they do not focus on eye tracking studies for analysing eye movement sequences of users (i.e., scanpaths) on web pages. We investigate the effects of the number of users on scanpath analysis with our algorithm that was designed for identifying the most commonly followed path by multiple users. Our experimental results suggest that it is possible to approximate the same results with a smaller number of users. The results also suggest that more users are required when they serendipitously browse on web pages in comparison with when they search for specific information or items. We observed that we could achieve 75% similarity to the results of 65 users with 27 users for searching tasks and 34 users for browsing tasks. This study guides researchers to determine the ideal number of users for analysing scanpaths on web pages based on their budget and time.

[1]  Gavriel Salvendy,et al.  Number of people required for usability evaluation , 2010, Commun. ACM.

[2]  James R. Lewis.  Testing Small System Customer Set-Up , 1982 .

[3]  Yeliz Yesilada,et al.  Patterns in Eyetracking Scanpaths and the Affecting Factors , 2015, J. Web Eng..

[4]  James R. Lewis,et al.  Evaluation of Procedures for Adjusting Problem-Discovery Rates Estimated From Small Samples , 2001, Int. J. Hum. Comput. Interact..

[5]  Joseph H. Goldberg,et al.  Scanpath clustering and aggregation , 2010, ETRA.

[6]  Geoffrey M. Underwood,et al.  Knowledge-Based Patterns of Remembering: Eye Movement Scanpaths Reflect Domain Experience , 2008, USAB.

[7]  Haruhiko Takeuchi,et al.  A Quantitative Method for Analyzing Scan Path Data Obtained by Eye Tracker , 2007, 2007 IEEE Symposium on Computational Intelligence and Data Mining.

[8]  Anja Janoschka,et al.  Web Advertising: New Forms of Communication on the Internet , 2004 .

[9]  Ebba Þóra Hvannberg,et al.  Analysis of combinatorial user effect in international usability tests , 2004, CHI '04.

[10]  M. Elgin Akpinar,et al.  Vision Based Page Segmentation Algorithm: Extended and Perceived Success , 2013, ICWE Workshops.

[11]  Jakob Nielsen,et al.  The "magic number 5": is it enough for web testing? , 2002, CHI Extended Abstracts.

[12]  Hervé Abdi,et al.  Children use visual speech to compensate for non-intact auditory speech. , 2014, Journal of experimental child psychology.

[13]  Jan Theeuwes,et al.  ScanMatch: A novel method for comparing fixation sequences , 2010, Behavior research methods.

[14]  Anne R. Haake,et al.  eyePatterns: software for identifying patterns and similarities across fixation sequences , 2006, ETRA.

[15]  Jared M. Spool,et al.  Testing web sites: five users is nowhere near enough , 2001, CHI Extended Abstracts.

[16]  Martin Schmettow,et al.  Sample size in usability studies , 2012, Commun. ACM.

[17]  M. Elgin Akpinar,et al.  "Old habits die hard!": eyetracking based experiential transcoding: a study with mobile users , 2015, W4A.

[18]  Pam J. Mayhew,et al.  How many participants are really enough for usability studies? , 2014, 2014 Science and Information Conference.

[19]  Whitepaper,et al.  Using the Tobii Mobile Device Stand in Usability Testing on Mobile Devices , 2012 .

[20]  Yeliz Yesilada,et al.  Scanpath Trend Analysis on Web Pages , 2016, ACM Trans. Web.

[21]  Gary Marchionini,et al.  Exploratory search , 2006, Commun. ACM.

[22]  Yeliz Yesilada,et al.  Eye tracking scanpath analysis techniques on web pages: A survey, evaluation and comparison , 2015 .

[23]  Eleni Michailidou,et al.  ViCRAM: visual complexity rankings and accessibility metrics , 2006, ASAC.

[24]  Yeliz Yesilada,et al.  Identifying Patterns in Eyetracking Scanpaths in Terms of Visual Elements of Web Pages , 2014, ICWE.

[25]  Melanie Volkamer,et al.  Building castles in quicksand: blueprint for a crowdsourced study , 2014, CHI Extended Abstracts.

[26]  Jakob Nielsen,et al.  A mathematical model of the finding of usability problems , 1993, INTERCHI.

[27]  Qi Zhao,et al.  Webpage Saliency , 2014, ECCV.

[28]  Thierry Baccino,et al.  New insights into ambient and focal visual fixations using an automatic classification algorithm , 2011, i-Perception.

[29]  SalvendyGavriel,et al.  Number of people required for usability evaluation , 2010 .

[30]  Boris M. Velichkovsky,et al.  Towards an express-diagnostics for level of processing and hazard perception , 2002 .

[31]  L. Faulkner.  Beyond the five-user assumption: Benefits of increased sample sizes in usability testing , 2003, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[32]  Michael E. Holmes,et al.  Visual attention to repeated internet images: testing the scanpath theory on the world wide web , 2002, ETRA.

[33]  Yeliz Yesilada,et al.  Experiential transcoding: an EyeTracking approach , 2013, W4A.

[34]  Stephanie Wilson,et al.  Identifying web usability problems from eye-tracking data , 2007, BCS HCI.