Usability Engineering for the Adaptive Web

This chapter discusses a usability engineering approach for the design and the evaluation of adaptive web-based systems, focusing on practical issues. A list of methods will be presented, considering a user-centered approach. After having introduced the peculiarities that characterize the evaluation of adaptive web-based systems, the chapter describes the evaluation methodologies following the temporal phases of evaluation, according to a user-centered approach. Three phases are distinguished: requirement phase, preliminary evaluation phase, and final evaluation phase. Moreover, every technique is classified according to a set of parameters that highlight the practical exploitation of that technique. For every phase, the appropriate techniques are described by giving practical examples of their application in the adaptive web. A number of issues that arise when evaluating an adaptive system are described, and potential solutions and workarounds are sketched.

[1]  A. Jameson.  Adaptive interfaces and agents , 2002 .

[2]  Anselm L. Strauss,et al.  Basics of qualitative research : techniques and procedures for developing grounded theory , 1998 .

[3]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[4]  Stephan Weibelzahl,et al.  Evaluating the Inference Mechanism of Adaptive Learning Systems , 2003, User Modeling.

[5]  Federica Cena,et al.  How to Communicate Recommendations? Evaluation of an Adaptive Annotation Technique , 2005, INTERACT.

[6]  Ben Shneiderman,et al.  Split menus: effectively using selection frequency to organize menus , 1994, TCHI.

[7]  Kristina Höök,et al.  Evaluating the utility and usability of an adaptive hypermedia system , 1997, IUI '97.

[8]  John R. Anderson,et al.  The Geometry Tutor , 1985, IJCAI.

[9]  Carlo Strapparava,et al.  Adaptive Hypermedia and Adaptive Web-Based Systems, 5th International Conference, AH 2008, Hannover, Germany, July 29 - August 1, 2008. Proceedings , 2008, AH.

[10]  Alfred Kobsa,et al.  The Adaptive Web, Methods and Strategies of Web Personalization , 2007, The Adaptive Web.

[11]  Brian Shackel,et al.  Human factors for informatics usability , 1991 .

[12]  T. Landauer,et al.  Handbook of Human-Computer Interaction , 1997 .

[13]  Francesco Ricci,et al.  Understanding Recommender Systems : Experimental Evaluation Challenges , 2003 .

[14]  Fabio Paternò,et al.  Human-Computer Interaction - INTERACT 2005 , 2005, Lecture Notes in Computer Science.

[15]  Antonio Krüger,et al.  Adaptive Mobile Guides , 2007, The Adaptive Web.

[16]  Paul Brna,et al.  User Modeling 2005, 10th International Conference, UM 2005, Edinburgh, Scotland, UK, July 24-29, 2005, Proceedings , 2005, User Modeling.

[17]  Roman Bednarik,et al.  Potentials of Eye-Movement Tracking in Adaptive Systems , 2009 .

[18]  Jack Mostow,et al.  Assessing Student Proficiency in a Reading Tutor That Listens , 2003, User Modeling.

[19]  Eelco Herder.  Utility-Based Evaluation of Adaptive Systems , 2003 .

[20]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[21]  Shimei Pan,et al.  Predicting and Adapting to Poor Speech Recognition in a Spoken Dialogue System , 2000, AAAI/IAAI.

[22]  G. D. Magoulas,et al.  Integrating Layered and Heuristic Evaluation for Adaptive Learning Environments , 2003 .

[23]  S. Fagerhaugh,et al.  Participant Observation , 1979 .

[24]  Franck Tarpin-Bernard,et al.  Adaptive Presentation of Multimedia Interface Case Study: "Brain Story" Course , 2003, User Modeling.

[25]  Alex Paramythis,et al.  A Decomposition Model for the Layered Evaluation of Interactive Adaptive Systems , 2005, User Modeling.

[26]  John L. Bennett,et al.  Usability Engineering: Our Experience and Evolution , 1988 .

[27]  Peter Brusilovsky,et al.  A Study of User Model Based Link Annotation in Educational Hypermedia , 1998, J. Univers. Comput. Sci..

[28]  M. Gardner,et al.  Statistical guidelines for contributors to medical journals. , 1983, British medical journal.

[29]  C. M. Tobar.  Yet Another Evaluation Framework , 2003 .

[30]  Lachlan M. MacKinnon,et al.  User Cognitive Style and Interface Design for Personal, Adaptive Learning. What to Model? , 2005, User Modeling.

[31]  Bamshad Mobasher,et al.  Data Mining for Web Personalization , 2007, The Adaptive Web.

[32]  Arthur C. Graesser,et al.  Modeling Student Performance to Enhance the Pedagogy of AutoTutor , 2003, User Modeling.

[33]  Tsvi Kuflik,et al.  Privacy Preservation Improvement by Learning Optimal Profile Generation Rate , 2003, User Modeling.

[34]  David Benyon,et al.  Adaptive systems: A solution to usability problems , 1993, User Modeling and User-Adapted Interaction.

[35]  Nandish V. Patel.  Adaptive Evolutionary Information Systems , 2002 .

[36]  Christian A. Müller,et al.  Recognizing Time Pressure and Cognitive Load on the Basis of Speech: An Experimental Study , 2001, User Modeling.

[37]  BrittonCarol,et al.  The Use of a Co-operative Student Model of Learner Characteristics to Configure a Multimedia Application , 2002 .

[38]  G. Keppel,et al.  Introduction to Design and Analysis : A Student's Handbook , 1980 .

[39]  Yumiko Hara,et al.  Categorization of Japanese TV Viewers Based on Program Genres They Watch , 2004, User Modeling and User-Adapted Interaction.

[40]  Bradley N. Miller,et al.  Using filtering agents to improve prediction quality in the GroupLens research collaborative filtering system , 1998, CSCW '98.

[41]  Michael J. Pazzani,et al.  Learning Collaborative Information Filters , 1998, ICML.

[42]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[43]  Cristina Gena,et al.  Evaluation of an on-vehicle adaptive tourist service , 2003 .

[44]  Alexandros Paramythis,et al.  A modular approach to the evaluation of Adaptive User Interfaces , 2001 .

[45]  Liliana Ardissono,et al.  Intelligent Support to the Retrieval of Information About Hydric Resources , 2004, AH.

[46]  Frank E. Ritter,et al.  Techniques for Modeling Human Performance in Synthetic Environments: A Supplementary Review , 2003 .

[47]  Alison Cawsey,et al.  Does Adapted Information Help Patients with Cancer? , 2003, User Modeling.

[48]  Michael J. Pazzani,et al.  Content-Based Recommendation Systems , 2007, The Adaptive Web.

[49]  Martha E. Crosby,et al.  A Model for Integrating an Adaptive Information Filter Utilizing Biosensor Data to Assess Cognitive Load , 2003, User Modeling.

[50]  Josef F. Krems,et al.  Learning Grammar with Adaptive Hypertexts: Reading or Searching? , 2002, AH.

[51]  Sean M. McNee,et al.  Interfaces for Eliciting New User Preferences in Recommender Systems , 2003, User Modeling.

[52]  Anthony Jameson,et al.  Making systems sensitive to the user's changing resource limitations , 1999, Knowl. Based Syst..

[53]  Tom Carey,et al.  Human-computer interaction , 1994 .

[54]  Marek Czarkowski.  Evaluating Scrutable Adaptive Hypertext , 2005 .

[55]  Judith Masthoff.  The evaluation of adaptive systems , 2003 .

[56]  Antonija Mitrovic,et al.  Evaluating the Effects of Open Student Models on Learning , 2002, AH.

[57]  Andrea Bunt,et al.  User Modelling to Support User Customization , 2005, User Modeling.

[58]  Stephan Weibelzahl,et al.  A Database of Empirical Evaluations of Adaptive Systems , 2001 .

[59]  Peter Brusilovsky,et al.  User Models for Adaptive Hypermedia and Adaptive Educational Systems , 2007, The Adaptive Web.

[60]  Anthony Jameson,et al.  User Modeling Meets Usability Goals , 2005, User Modeling.

[61]  Stephan Weibelzahl.  Problems and Pitfalls in the Evaluation of Adaptive Systems , 2005 .

[62]  Peter Brusilovsky,et al.  Addictive Links: The Motivational Value of Adaptive Link Annotation in Educational Hypermedia , 2006, AH.

[63]  Gregory D. Abowd,et al.  Human-computer interaction (2nd ed.) , 1998 .

[64]  T. Lang,et al.  How to Report Statistics in Medicine: Annotated Guidelines for Authors , 1997 .

[65]  Fabio Abbattista,et al.  Learning Interaction Models in a Digital Library Service , 2001, User Modeling.

[66]  Martin R. Gibbs,et al.  Mediating intimacy: designing technologies to support strong-tie relationships , 2005, CHI.

[67]  Morten Kyng,et al.  Design at Work , 1992 .

[68]  Cristina Gena,et al.  Methods and techniques for the evaluation of user-adaptive systems , 2005, The Knowledge Engineering Review.

[69]  Constantine Stephanidis,et al.  # 2001 Kluwer Academic Publishers. Printed in the Netherlands. Adaptive Techniques for Universal Access , 1999 .

[70]  Debora Shaw,et al.  Handbook of usability testing: How to plan, design, and conduct effective tests , 1996 .

[71]  Ning Wang,et al.  A Semi-automated Wizard of Oz Interface for Modeling Tutorial Strategies , 2005, User Modeling.

[72]  Giacomo Mauro DAriano.  The Handbook for Focus Group Research. , 1994 .

[73]  Wai-Tat Fu,et al.  SNIF-ACT: A Model of Information Foraging on the World Wide Web , 2003, User Modeling.

[74]  Amruth Kumar Ramapo.  Evaluating Adaptive Generation of Problems in Programming Tutors – Two Studies , 2006 .

[75]  Paul Brna,et al.  Applying Interactive Open Learner Models to Learning Technical Terminology , 2001, User Modeling.

[76]  S. Floyd,et al.  Adaptive Web , 1997 .

[77]  Tom Lang,et al.  How to Report Statistics in Medicine: Annotated Guidelines for Authors, Editors, and Reviewers , 1997 .

[78]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[79]  Wolfgang Pohl,et al.  LaboUr - Machine Learning for User Modeling , 1997, HCI.

[80]  Eelco Herder,et al.  Much to Know About History , 2006, AH.

[81]  Cristina Gena,et al.  THE IMPORTANCE OF ADAPTIVITY TO PROVIDE ONBOARD SERVICES: A PRELIMINARY EVALUATION OF AN ADAPTIVE TOURIST INFORMATION SERVICE ONBOARD VEHICLES , 2004, Appl. Artif. Intell..

[82]  G. Travassos,et al.  Contributions of In Virtuo and In Silico Experiments for the Future of Empirical Studies in Software Engineering , 2003 .

[83]  Vania Dimitrova.  Using Dialogue Games to Maintain Diagnostic Interactions , 2003, User Modeling.

[84]  Maria Virvou,et al.  User Modelling: An Empirical Study for Affect Perception Through Keyboard and Speech in a Bi-modal User Interface , 2006, AH.

[85]  G. Keppel,et al.  Design and Analysis: A Researcher's Handbook , 1976 .

[86]  Antonija Mitrovic.  Investigating Students' Self-Assessment Skills , 2001, User Modeling.

[87]  Jintae Lee,et al.  What's in Design Rationale? , 1991, Hum. Comput. Interact..

[88]  Liliana Ardissono,et al.  Personalization in E-Commerce Applications , 2007, The Adaptive Web.

[89]  Alfred Kobsa,et al.  Adaptable and adaptive information provision for all users, including disabled and elderly people , 1998, New Rev. Hypermedia Multim..

[90]  Antonija Mitrovic,et al.  WETAS: A Web-Based Authoring System for Constraint-Based ITS , 2002, AH.

[91]  Georgios Paliouras,et al.  Web Usage Mining as a Tool for Personalization: A Survey , 2003, User Modeling and User-Adapted Interaction.

[92]  A. Sockloff,et al.  Statistical power analysis for the behavioral sciences: (revised edition), by Jacob Cohen. New York: Academic Press, 1977, xv + 474 pp., $24.50. , 1978 .

[93]  Joseph E. Beck,et al.  Using the Student Model to Control Problem Difficulty , 1997 .

[94]  Gregory D. Abowd,et al.  Human-Computer Interaction. Second Edition. , 1998 .

[95]  Leland Wilkinson,et al.  Statistical Methods in Psychology Journals Guidelines and Explanations , 2005 .

[96]  Peter Brusilovsky,et al.  The Benefits of Layered Evaluation of Adaptive Applications and Services , 2001 .

[97]  Sara Jones,et al.  The Use of a Co-operative Student Model of Learner Characteristics to Configure a Multimedia Application , 2002, User Modeling and User-Adapted Interaction.

[98]  Catherine Courage,et al.  Understanding Your Users: A Practical Guide to User Requirements Methods, Tools, and Techniques , 2005 .

[99]  Lambertus J.M. Mulder,et al.  Human Factors in Transportation, Communication, Health, and the Workplace. , 2002 .

[100]  Jean Scholtz,et al.  VISVIP: 3D visualization of paths through web sites , 1999, Proceedings. Tenth International Workshop on Database and Expert Systems Applications. DEXA 99.

[101]  Paul Resnick,et al.  Recommender systems , 1997, CACM.

[102]  Judy Kay,et al.  Iems: Helping Users Manage Email , 2003, User Modeling.

[103]  James C. Lester,et al.  Cinematographic User Models for Automated Realtime Camera Control in Dynamic 3D Environments , 1997 .

[104]  Pattie Maes,et al.  Social information filtering: algorithms for automating “word of mouth” , 1995, CHI '95.

[105]  Eugene Santos,et al.  Empirical Evaluation of Adaptive User Modeling in a Medical Information Retrieval Application , 2003, User Modeling.

[106]  Gena Cristina,et al.  E-tool: a personalised prototype for web based applications , 2002 .

[107]  Karen Holtzblatt,et al.  Contextual design , 1997, INTR.

[108]  Egon Berghout,et al.  The Goal/Question/Metric method: a practical guide for quality improvement of software development , 1999 .

[109]  Paul P. Maglio,et al.  GAS: Group Adaptive System , 2002, AH.

[110]  Myra Spiliopoulou,et al.  Web usage mining for Web site evaluation , 2000, CACM.

[111]  Steve Krug,et al.  Don't Make Me Think!: A Common Sense Approach to Web Usability , 2000 .

[112]  S. Weibelzahl,et al.  Framework for the Evaluation of Adaptive CBR-Systems , 2001 .

[113]  Piotr J. Gmytrasiewicz,et al.  User Modeling 2001 , 2001, Lecture Notes in Computer Science.

[114]  Dan Diaper,et al.  Task Analysis for Human-Computer Interaction , 1990 .

[115]  Yutaka Matsuo.  Word Weighting Based on User's Browsing History , 2003, User Modeling.

[116]  Kasia Muldner,et al.  Exploring Eye Tracking to Increase Bandwidth in User Modeling , 2005, User Modeling.

[117]  Alfred Kobsa,et al.  Interaction of domain expertise and interface design in adaptive educational hypermedia , 1999 .

[118]  Christoph G. Thomas,et al.  A user interface integrating adaptability and adaptivity , 1994 .

[119]  Susan Bull,et al.  User Modelling and Mobile Learning , 2003, User Modeling.

[120]  David N. Chin.  Empirical Evaluation of User Models and User-Adapted Systems , 2001, User Modeling and User-Adapted Interaction.

[121]  Enrique Alfonseca,et al.  Modelling Users' Interests and Needs for an Adaptive Online Information System , 2003, User Modeling.

[122]  George D. Magoulas,et al.  Adaptable and Adaptive Hypermedia Systems , 2005 .

[123]  Peter Brusilovsky,et al.  Social Navigation Support in a Course Recommendation System , 2006, AH.

[124]  Cristina Gena.  Evaluation methodologies and user involvement in user modeling and adaptive systems , 2003 .

[125]  Antonija Mitrovic,et al.  The Effect of Adapting Feedback Generality in ITS , 2006, AH.

[126]  Fabio Bellifemine,et al.  User Modeling and Recommendation Techniques for Personalized Electronic Program Guides , 2004, Personalized Digital Television.

[127]  Oren Etzioni,et al.  Adaptive Web sites , 2000, CACM.

[128]  John Riedl,et al.  Combining Collaborative Filtering with Personal Agents for Better Recommendations , 1999, AAAI/IAAI.

[129]  Stephan Weibelzahl,et al.  Evaluation of Adaptive Systems , 2001, User Modeling.

[130]  Patrick Baudisch,et al.  TV Scout: Lowering the Entry Barrier to Personalized TV Program Recommendation , 2002, From Integrated Publication and Information Systems to Virtual Information and Knowledge Environments.

[131]  Joachim Meyer,et al.  The Evaluation of In-Vehicle Adaptive Systems , 2005 .

[132]  Saul Greenberg,et al.  Prototyping an intelligent agent through Wizard of Oz , 1993, INTERCHI.

[133]  Ion Juvina,et al.  The Impact of Link Suggestions on User Navigation and User Perception , 2005, User Modeling.

[134]  Stephan Weibelzahl,et al.  Eliciting Requirements for a Adaptive Decision SupportSystem through Structured User Interviews , 2006 .

[135]  Jaideep Srivastava,et al.  Automatic personalization based on Web usage mining , 2000, CACM.

[136]  Barry Smyth,et al.  Personalized Adaptive Navigation for Mobile Portals , 2002, ECAI.

[137]  I. Olkin,et al.  Improving the quality of reporting of randomized controlled trials. The CONSORT statement. , 1996, JAMA.

[138]  Jack Mostow,et al.  Predicting Student Help-Request Behavior in an Intelligent Tutor for Reading , 2003, User Modeling.

[139]  Cristina Gena,et al.  Designing TV Viewer Stereotypes for an Electronic Program Guide , 2001, User Modeling.

[140]  Alphonse Chapanis.  Evaluating usability , 1991 .

[141]  Robert K. Young,et al.  Introduction to design and analysis: A student's handbook. , 1980 .

[143]  Robin D. Burke,et al.  Hybrid Web Recommender Systems , 2007, The Adaptive Web.

[144]  Markus A. Thies.  Adaptive User Interfaces , 1994, IFIP Congress.

[145]  N. Hoffart.  Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory , 2000 .

[146]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[147]  Peter Brusilovsky,et al.  Adaptive Hypermedia: From Intelligent Tutoring Systems to Web-Based Education , 2000, Intelligent Tutoring Systems.

[148]  Kristina Höök,et al.  Steps to take before intelligent user interfaces become real , 2000, Interact. Comput..

[149]  Joseph F. Dumas,et al.  A Practical Guide to Usability Testing , 1993 .

[150]  Fabio Paternò,et al.  Intelligent analysis of user interactions with web applications , 2002, IUI '02.

[151]  Anthony Jameson,et al.  Pros and Cons of Controllability: An Empirical Study , 2002, AH.

[152]  Dietmar Jannach,et al.  Special issue on Recommender Systems , 2008, AI Commun..

[153]  Gerhard Fischer,et al.  Personalizing Delivered Information in a Software Reuse Environment , 2001, User Modeling.

[154]  Demetrios G. Sampson,et al.  Layered Evaluation of Adaptive Applications and Services , 2000, AH.

[155]  Judith Masthoff,et al.  Proceedings of the Fifth Workshop on User-Centred Design and Evaluation of Adaptive Systems, associated with AH'06 , 2006 .

[156]  Carlo Strapparava,et al.  Improving User Modelling with Content-Based Techniques , 2001, User Modeling.

[157]  Judith Masthoff,et al.  Workshop on User-Centred Design and Evaluation of Adaptive Systems , 2009 .

[158]  Judy Kay,et al.  Learner Control , 2001, User Modeling and User-Adapted Interaction.

[159]  Alfred Kobsa.  User Modeling and User-Adapted Interaction , 2005, User Modeling and User-Adapted Interaction.

[160]  Frank Van Harmelen.  ECAI 2002 : 15th European Conference on Artificial Intelligence, July 21-26, 2002, Lyon,France : including Prestigious Applications of Intelligent Systems (PAIS 2002) : proceedings , 2002 .

[161]  Peter Brusilovsky,et al.  Adaptive Navigation Support in Educational Hypermedia : An Evaluation of the ISIS-Tutor , 1998 .

[162]  Petri Nokelainen,et al.  EDUCO - A Collaborative Learning Environment Based on Social Navigation , 2002, AH.

[163]  Peter Brusilovsky,et al.  From adaptive hypermedia to the adaptive web , 2002, CACM.

[164]  Stanislav L. Stoev,et al.  An application-independent intelligent user support system exploiting action-sequence based user modelling , 1999 .

[165]  Robin D. Burke,et al.  Hybrid Recommender Systems: Survey and Experiments , 2002, User Modeling and User-Adapted Interaction.

[166]  Peter Brusilovsky,et al.  User Modeling 2003 , 2003, Lecture Notes in Computer Science.