Automatic detection of users’ skill levels using high-frequency user interface events

Computer users have different levels of system skills. Moreover, each user has different levels of skill across different applications and even in different portions of the same application. Additionally, users’ skill levels change dynamically as users gain more experience in a user interface. In order to adapt user interfaces to the different needs of user groups with different levels of skills, automatic methods of skill detection are required. In this paper, we present our experiments and methods, which are used to build automatic skill classifiers for desktop applications. Machine learning algorithms were used to build statistical predictive models of skill. Attribute values were extracted from high frequency user interface events, such as mouse motions and menu interactions, and were used as inputs to our models. We have built both task-independent and task-dependent classifiers with promising results.

[1]  Matthias Schneider-Hufschmidt,et al.  Adaptive User Interfaces: Principles and Practice , 1993 .

[2]  E. R. Crossman.  A THEORY OF THE ACQUISITION OF SPEED-SKILL∗ , 1959 .

[3]  Arbi Ghazarian,et al.  A Case Study of Defect Introduction Mechanisms , 2009, CAiSE.

[4]  Eli Hagen,et al.  Adaptation of an automotive dialogue system to users’ expertise and evaluation of the system , 2006, SIGDIAL.

[5]  Kent P. Vaubel,et al.  Inferring User Expertise for Adaptive Interfaces , 1990, Hum. Comput. Interact..

[6]  Jonathan Grudin,et al.  Human Computer Interaction: The Year 2000 and Beyond , 1995, HCI.

[7]  Ian Witten,et al.  Data Mining , 2000 .

[8]  Geoffrey I. Webb,et al.  Using Decision Trees for Agent Modeling: Improving Prediction Performance , 2004, User Modeling and User-Adapted Interaction.

[9]  Gregory D. Hager,et al.  Towards automatic skill evaluation: detection and segmentation of robot-assisted surgical motions. , 2006 .

[10]  Jonathan Kies,et al.  User and task analysis for interface design , 1998 .

[11]  L. Miguel Encarnação.  Multi-level user support through adaptive hypermedia: a highly application-independent help component , 1997, IUI '97.

[12]  Paul M. Fitts,et al.  Perceptual-Motor Skill Learning1 , 1964 .

[13]  Gary M. Olson,et al.  The growth of cognitive modeling in human-computer interaction since GOMS , 1990 .

[14]  Sher ry Folsom-Meek,et al.  Human Performance , 2020, Nature.

[15]  Peter Brusilovsky,et al.  Adaptive Hypermedia , 2001, User Modeling and User-Adapted Interaction.

[16]  A. Darzi,et al.  The use of electromagnetic motion tracking analysis to objectively measure open surgical skill in the laboratory-based model. , 2001, Journal of the American College of Surgeons.

[17]  Woodrow Barfield,et al.  Expert-novice differences for software: implications for problem-solving and knowledge acquisition , 1986 .

[18]  Maxwel Roberts,et al.  Integrating the Mind , 2007 .

[19]  H. Simon,et al.  A simulation of memory for chess positions. , 1973 .

[20]  John R. Anderson.  The Architecture of Cognition , 1983 .

[21]  Ingrid Zukerman,et al.  # 2001 Kluwer Academic Publishers. Printed in the Netherlands. Predictive Statistical Models for User Modeling , 1999 .

[22]  John R. Anderson.  Acquisition of cognitive skill. , 1982 .

[23]  Scott E. Hudson,et al.  Dynamic detection of novice vs. skilled use without a task model , 2007, CHI.

[24]  Albert N. Badre,et al.  Automatic chunk detection in human-computer interaction , 1994, AVI '94.

[25]  Gerhard Fischer,et al.  User Modeling in Human–Computer Interaction , 2001, User Modeling and User-Adapted Interaction.

[26]  Penelope M. Sanderson,et al.  Exploratory Sequential Data Analysis: Foundations , 1994, Hum. Comput. Interact..

[27]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[28]  Elaine Rich.  Users are individuals: individualizing user models , 1999, Int. J. Hum. Comput. Stud..

[29]  Allen and Rosenbloom Paul S. Newell,et al.  Mechanisms of Skill Acquisition and the Law of Practice , 1993 .

[30]  Xiaohui Liu,et al.  The role of human factors in stereotyping behavior and perception of digital library users: a robust clustering approach , 2007, User Modeling and User-Adapted Interaction.

[31]  Kristiina Jokinen,et al.  User Expertise Modeling and Adaptivity in a Speech-Based E-Mail System , 2004, ACL.

[32]  Herbert A. Simon,et al.  Applications of machine learning and rule induction , 1995, CACM.

[33]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. 1954. , 1992, Journal of experimental psychology. General.

[34]  Markus A. Thies.  Adaptive User Interfaces , 1994, IFIP Congress.

[35]  John Fulcher,et al.  Classification of User Expertise Level by Neural Networks , 1997, Int. J. Neural Syst..

[36]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[37]  A. Darzi,et al.  Skills assessment of surgeons. , 2002, Surgery.

[38]  FischerGerhard.  User Modeling in HumanComputer Interaction , 2001 .

[39]  Jens Rasmussen,et al.  Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models , 1983, IEEE Transactions on Systems, Man, and Cybernetics.

[40]  David E. Kieras,et al.  Using GOMS for user interface design and evaluation: which technique? , 1996, TCHI.

[41]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[42]  Peter Brusilovsky,et al.  Methods and techniques of adaptive hypermedia , 1996, User Modeling and User-Adapted Interaction.

[43]  H. Simon,et al.  Perception in chess , 1973 .

[44]  A. Darzi,et al.  Relationship between skill and outcome in the laboratory-based model. , 2002, Surgery.

[45]  David F. Redmiles,et al.  Extracting usability information from user interface events , 2000, CSUR.

[46]  Alfred Kobsa,et al.  Generic User Modeling Systems , 2001, User modeling and user-adapted interaction.

[47]  T J Triggs,et al.  Characteristics of cursor trajectories controlled by the computer mouse , 2001, Ergonomics.

[48]  Blake Hannaford,et al.  Markov modeling of minimally invasive surgery based on tool/tissue interaction and force/torque signatures for evaluating surgical skills , 2001, IEEE Transactions on Biomedical Engineering.

[49]  David Benyon,et al.  Adaptive systems: A solution to usability problems , 1993, User Modeling and User-Adapted Interaction.

[50]  B. Hannaford,et al.  Skills evaluation in minimally invasive surgery using force/torque signatures , 2000, Surgical Endoscopy.

[51]  P. Fitts.  The information capacity of the human motor system in controlling the amplitude of movement. , 1954, Journal of experimental psychology.

[52]  Kirk P. Arnett,et al.  Performance Effect of Matching Computer Interface Characteristics and User Skill Level , 1993, Int. J. Man Mach. Stud..

[53]  Aiko M. Hormann,et al.  Programs for Machine Learning. Part I , 1962, Inf. Control..

[54]  A. W. Melton.  Categories of Human Learning , 1964 .

[55]  James D. Foley,et al.  Supporting adaptive interfaces in a knowledge-based user interface environment , 1993, IUI '93.

[56]  M. Brewer,et al.  Research Design and Issues of Validity , 2000 .

[57]  Karl Wolf,et al.  User action graphing effort (UsAGE) , 1995, CHI '95.

[58]  David N. Chin.  KNOME: Modeling What the User Knows in UC , 1989 .

[59]  Geoffrey I. Webb,et al.  # 2001 Kluwer Academic Publishers. Printed in the Netherlands. Machine Learning for User Modeling , 1999 .

[60]  C. Lee Giles,et al.  Sequence learning: from recognition and prediction to sequential decision making , 2001, IEEE Intelligent Systems.

[61]  Paul Bratley,et al.  Computers and the Humanities , 1978, Computer.

[62]  Ron Kohavi,et al.  The Case against Accuracy Estimation for Comparing Induction Algorithms , 1998, ICML.

[63]  A. Kobsa.  User Modeling : Recent Work , Prospects and Hazards , 1993 .

[64]  Jason E. Robbins,et al.  Supporting Ongoing User Involvement in Development via Expectation-Driven Event Monitoring , 2007 .

[65]  Jiming Liu,et al.  Exploring the applications of user-expertise assessment for intelligent interfaces , 1993, CHI '93.

[66]  David Wick,et al.  Cross-user analysis: Benefits of skill level comparison in usability testing , 2005, Interact. Comput..

[67]  John R. Anderson,et al.  Knowledge Compilation: Mechanisms for the Automatization of Cognitive Skills. , 1980 .

[68]  Henry C. Lin,et al.  Towards automatic skill evaluation: Detection and segmentation of robot-assisted surgical motions , 2006, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[69]  Allen Newell,et al.  The keystroke-level model for user performance time with interactive systems , 1980, CACM.

[70]  Rich Caruana,et al.  Multitask Learning , 1997, Machine-mediated learning.

[71]  Eric Horvitz,et al.  The Lumière Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users , 1998, UAI.

[72]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.