Design and evaluation of a command recommendation system for software applications

We examine the use of modern recommender system technology to aid command awareness in complex software applications. We first describe our adaptation of traditional recommender system algorithms to meet the unique requirements presented by the domain of software commands. A user study showed that our item-based collaborative filtering algorithm generates 2.1 times as many good suggestions as existing techniques. Motivated by these positive results, we propose a design space framework and its associated algorithms to support both global and contextual recommendations. To evaluate the algorithms, we developed the CommunityCommands plug-in for AutoCAD. This plug-in enabled us to perform a 6-week user study of real-time, within-application command recommendations in actual working environments. We report and visualize command usage behaviors during the study, and discuss how the recommendations affected users behaviors. In particular, we found that the plug-in successfully exposed users to new commands, as unique commands issued significantly increased.

[1]  Ian H. Witten,et al.  On Frequency-Based Menu-Splitting Algorithms , 1984, Int. J. Man Mach. Stud..

[2]  John Riedl,et al.  An Empirical Analysis of Design Choices in Neighborhood-Based Collaborative Filtering Algorithms , 2002, Information Retrieval.

[3]  John F. Canny,et al.  Collaborative filtering with privacy via factor analysis , 2002, SIGIR '02.

[4]  L. A. Hermens,et al.  A machine-learning apprentice for the completion of repetitive forms , 1993, Proceedings of 9th IEEE Conference on Artificial Intelligence for Applications.

[5]  Kevin Knabe.  Apple guide: a case study in user-aided design of online help , 1995, CHI '95.

[6]  George Kingsley Zipf,et al.  Human Behaviour and the Principle of Least Effort: an Introduction to Human Ecology , 2012 .

[7]  Mary Czerwinski,et al.  Effects of instant messaging interruptions on computing tasks , 2000, CHI Extended Abstracts.

[8]  John Riedl,et al.  Item-based collaborative filtering recommendation algorithms , 2001, WWW '01.

[9]  David Heckerman,et al.  Empirical Analysis of Predictive Algorithms for Collaborative Filtering , 1998, UAI.

[10]  Bradley N. Miller,et al.  MovieLens unplugged: experiences with an occasionally connected recommender system , 2003, IUI '03.

[11]  John Riedl,et al.  Analysis of recommendation algorithms for e-commerce , 2000, EC '00.

[12]  C. Trlica.  Software applications , 1996 .

[13]  Xue Li,et al.  Time weight collaborative filtering , 2005, CIKM '05.

[14]  Colin Potts,et al.  Studying the evolution and enhancement of software features , 2000, Proceedings 2000 International Conference on Software Maintenance.

[15]  Eric Horvitz,et al.  Principles of mixed-initiative user interfaces , 1999, CHI '99.

[16]  Tom M. Mitchell,et al.  A Personal Learning Apprentice , 1992, AAAI.

[17]  Kellogg S. Booth,et al.  An evaluation of a multiple interface design solution for bloated software , 2002, CHI.

[18]  Elizabeth D. Mynatt,et al.  Side views: persistent, on-demand previews for open-ended tasks , 2002, UIST '02.

[19]  Sean M. McNee,et al.  Improving recommendation lists through topic diversification , 2005, WWW '05.

[20]  Yehuda Koren,et al.  Collaborative filtering with temporal dynamics , 2009, KDD.

[21]  Gale Moore,et al.  Reducing the gap between what users know and what they need to know , 2000, CUU '00.

[22]  B. Shneiderman.  Promoting universal usability with multi-layer interface design , 2002, CUU '03.

[23]  Yi Zhang,et al.  Novelty and redundancy detection in adaptive filtering , 2002, SIGIR '02.

[24]  Brian P. Bailey,et al.  On the need for attention-aware systems: Measuring effects of interruption on task performance, error rate, and affective state , 2006, Comput. Hum. Behav..

[25]  Peter Brusilovsky,et al.  Social Navigation Support in a Course Recommendation System , 2006, AH.

[26]  Tovi Grossman,et al.  A survey of software learnability: metrics, methodologies and guidelines , 2009, CHI.

[27]  Òscar Celma,et al.  A new approach to evaluating novel recommendations , 2008, RecSys '08.

[28]  Yuen Ren Chao,et al.  Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology , 1950 .

[29]  Mary Czerwinski,et al.  Notification, Disruption, and Memory: Effects of Messaging Interruptions on Memory and Performance , 2001, INTERACT.

[30]  Mei-Hua Hsu,et al.  A personalized English learning recommender system for ESL students , 2008, Expert Syst. Appl..

[31]  Ben Shneiderman,et al.  Direct Manipulation: A Step Beyond Programming Languages , 1983, Computer.

[32]  LiWei,et al.  Design and evaluation of a command recommendation system for software applications , 2011 .

[33]  William W. Cohen,et al.  Recommendation as Classification: Using Social and Content-Based Information in Recommendation , 1998, AAAI/IAAI.

[34]  Eric Horvitz,et al.  The Lumière Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users , 1998, UAI.

[35]  Karen Spärck Jones.  A statistical interpretation of term specificity and its application in retrieval , 2021, J. Documentation.

[36]  Charles L. A. Clarke,et al.  Characterizing large-scale use of a direct manipulation application in the wild , 2010, Graphics Interface.

[37]  K. Gegenfurtner,et al.  Design Issues in Gaze Guidance Under review with ACM Transactions on Computer Human Interaction , 2009 .

[38]  Christopher G. Atkeson,et al.  Predicting human interruptibility with sensors: a Wizard of Oz feasibility study , 2003, CHI '03.

[39]  Jonathan L. Herlocker,et al.  Evaluating collaborative filtering recommender systems , 2004, TOIS.

[40]  Mark Rosenstein,et al.  Recommending and evaluating choices in a virtual community of use , 1995, CHI '95.

[41]  Joanna McGrenere,et al.  Beyond performance: Feature awareness in personalized interfaces , 2010, Int. J. Hum. Comput. Stud..

[42]  John T. Stasko,et al.  An empirical study of the effect of agent competence on user performance and perception , 2004, Proceedings of the Third International Joint Conference on Autonomous Agents and Multiagent Systems, 2004. AAMAS 2004..

[43]  Jiming Liu,et al.  An Adaptive User Interface Based On Personalized Learning , 2003, IEEE Intell. Syst..

[44]  Anna Wong,et al.  An adaptive calendar assistant using pattern mining for user preference modelling , 2010, IUI '10.

[45]  Frank Linton,et al.  Recommender Systems for Learning: Building User and Expert Models through Long-Term Observation of Application Use , 2000, User Modeling and User-Adapted Interaction.

[46]  T. Bösser.  Learning in Man-Computer Interaction: A Review of the Literature , 1987 .

[47]  Takeo Igarashi,et al.  A suggestive interface for 3D drawing , 2001, SIGGRAPH Courses.

[48]  Leah Kaufman,et al.  Too much of a good thing?: identifying and resolving bloat in the user interface , 1998, SGCH.

[49]  Mary Czerwinski,et al.  Introduction: design and evaluation of notification user interfaces , 2003, Int. J. Hum. Comput. Stud..

[50]  Brian D. Davison,et al.  Predicting Sequences of User Actions , 1998 .

[51]  Sean M. McNee,et al.  On the recommending of citations for research papers , 2002, CSCW '02.

[52]  Li Chen,et al.  Evaluating Critiquing-based Recommender Agents , 2006, AAAI.

[53]  Boi Faltings,et al.  Designing example-critiquing interaction , 2004, IUI '04.

[54]  Hiroshi Ishii,et al.  Ambient Displays: Turning Architectural Space into an Interface between People and Digital Information , 1998, CoBuild.

[55]  Robert Mack.  Understanding and learning text-editing skills: observations on the role of new user expectations , 1990 .

[56]  John M. Carroll,et al.  Training wheels in a user interface , 1984, CACM.

[57]  Tom Bösser.  Learning in Man-Computer Interaction , 1987 .

[58]  Francesco Ricci,et al.  Long-term and session-specific user preferences in a mobile recommender system , 2008, IUI '08.

[59]  Greg Linden,et al.  Amazon . com Recommendations Item-to-Item Collaborative Filtering , 2001 .

[60]  Sean M. McNee,et al.  Don't look stupid: avoiding pitfalls when recommending research papers , 2006, CSCW '06.

[61]  John Riedl,et al.  An algorithmic framework for performing collaborative filtering , 1999, SIGIR '99.

[62]  Mark Claypool,et al.  Implicit interest indicators , 2001, IUI '01.

[63]  John Riedl,et al.  GroupLens: an open architecture for collaborative filtering of netnews , 1994, CSCW '94.

[64]  Maria E. Orlowska,et al.  Recency-based collaborative filtering , 2006, ADC.

[65]  Tovi Grossman,et al.  CommunityCommands: command recommendations for software applications , 2009, UIST '09.

[66]  Gediminas Adomavicius,et al.  Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions , 2005, IEEE Transactions on Knowledge and Data Engineering.

[67]  Thomas Hofmann,et al.  Latent semantic models for collaborative filtering , 2004, TOIS.

[68]  Donald A. Norman,et al.  User Centered System Design: New Perspectives on Human-Computer Interaction , 1988 .

[69]  Sean M. McNee,et al.  Being accurate is not enough: how accuracy metrics have hurt recommender systems , 2006, CHI Extended Abstracts.

[70]  Richi Nayak,et al.  Improving Recommendation Novelty Based on Topic Taxonomy , 2007, 2007 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology - Workshops.

[71]  L. A. Hermens,et al.  A machine-learning apprentice for the completion of repetitive forms , 1994, IEEE Expert.

[72]  Gerhard Fischer,et al.  User Modeling in Human–Computer Interaction , 2001, User Modeling and User-Adapted Interaction.

[73]  Krzysztof Z. Gajos,et al.  Automatically generating personalized user interfaces with Supple , 2010, Artif. Intell..

[74]  Loren G. Terveen,et al.  Specifying preferences based on user history , 2002, CHI.