Automatically identifying targets users interact with during real world tasks

Information about the location and size of the targets that users interact with in real world settings can enable new innovations in human performance assessment and soft-ware usability analysis. Accessibility APIs provide some information about the size and location of targets. How-ever this information is incomplete because it does not sup-port all targets found in modern interfaces and the reported sizes can be inaccurate. These accessibility APIs access the size and location of targets through low-level hooks to the operating system or an application. We have developed an alternative solution for target identification that leverages visual affordances in the interface, and the visual cues produced as users interact with targets. We have used our novel target identification technique in a hybrid solution that combines machine learning, computer vision, and accessibility API data to find the size and location of targets users select with 89% accuracy. Our hybrid approach is superior to the performance of the accessibility API alone: in our dataset of 1355 targets covering 8 popular applications, only 74% of the targets were correctly identified by the API alone.

[1]  Peter Robinson,et al.  Cursor measures for motion-impaired computer users , 2002, ASSETS.

[2]  Simeon Keates,et al.  Developing steady clicks:: a method of cursor assistance for people with motor impairments , 2006, Assets '06.

[3]  Jean Ponce,et al.  Computer Vision: A Modern Approach , 2002 .

[4]  Gregory D. Abowd,et al.  Interaction techniques for ambiguity resolution in recognition-based interfaces , 2007, SIGGRAPH '07.

[5]  I. Scott MacKenzie,et al.  Accuracy measures for evaluating computer pointing devices , 2001, CHI.

[6]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[7]  Mark O. Riedl,et al.  A perception/action substrate for cognitive modeling in HCI , 2001, Int. J. Hum. Comput. Stud..

[8]  Olivier Chapuis,et al.  Fitts' Law in the Wild: A Field Study of Aimed Movements , 2007 .

[9]  Ian Witten,et al.  Data Mining , 2000 .

[10]  Rob Miller,et al.  Sikuli: using GUI screenshots for search and automation , 2009, UIST '09.

[11]  Joanna McGrenere,et al.  A comparison of static, adaptive, and adaptable menus , 2004, CHI.

[12]  Joyce Cox,et al.  Microsoft Office PowerPoint 2007 Step by Step , 2007 .

[13]  David Engel.  Microsoft Office Word 2007 Step By Step , 2007 .

[14]  Yoav Freund,et al.  Experiments with a New Boosting Algorithm , 1996, ICML.

[15]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[16]  Vimla L. Patel,et al.  Usability in the real world: assessing medical information technologies in patients' homes , 2003, J. Biomed. Informatics.

[17]  Eric Horvitz,et al.  Disruption and recovery of computing tasks: field study, analysis, and directions , 2007, CHI.

[18]  Shumin Zhai,et al.  More than dotting the i's --- foundations for crossing-based interfaces , 2002, CHI.

[19]  Luke S. Zettlemoyer,et al.  A visual medium for programmatic control of interactive applications , 1999, CHI '99.

[20]  Ian H. Witten,et al.  Data mining: practical machine learning tools and techniques, 3rd Edition , 1999 .

[21]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[22]  Henry Lieberman,et al.  Programming by example: visual generalization in programming by example , 2000, CACM.

[23]  Scott E. Hudson,et al.  Understanding pointing problems in real world computing environments , 2008, Assets '08.