Adoption-centric usability engineering: systematic deployment, evaluation and improvement of usability engineering methods in the software engineering lifecycle

Developing software systems which are easy to use and increase the productivity of users is still a major challenge in software engineering. Thus a large number of usability engineering methods have been proposed to systematically develop software with high usability. However studies indicate that even basic usability engineering methods are not integrated in software development lifecycles practiced in industrial settings. Research is required into the barriers of uptake of usability engineering methods within technically driven engineering disciplines. Yet problems in the adoption of methods by project teams are rarely examined. This work provides a new perspective on the problem of the uptake of usability engineering methods by software development teams. The adoption of methods by project teams contrary to common beliefs is not assured just because it is mandated by the organization. This work argues that usability engineering methods can only be regarded as integrated in the software development process of an organization, when these methods are practiced and accepted by development teams. So far no frameworks for examining the acceptance of methods by project teams and for exploiting such data for guiding project teams in method deployment are available. To address this problem this work presents an approach which consists of a process meta-model for guiding project teams in the deployment of usability engineering methods and a measurement framework for measuring the acceptance of the deployed methods. The approach is called adoption-centric usability engineering. The approach provides a concept to capture usability engineering methods together with a context profile. The context profile encodes the factors which contributed to the acceptance or rejection of the respective methods in past projects. This context-sensitive description of a usability engineering method introduced in this work is called a USEPack (Usability Engineering Experience Package). For guiding project teams in integrating usability engineering methods into their development processes the approach developed in this work introduces the concept of a usability engineering method kit. A usability engineering method kit is an abstraction of a usability engineering methodology which separates process phases and activities from the specific methods to perform them. The assignment of methods to certain activities is not predefined. They are chosen to best fit the characteristics of a project at hand. Usability engineering methods are linked to a usability engineering method kit in the form of USEPacks. The linking of USEPacks to a method kit is based on the characteristics of the project in which the method kit is to be deployed and on the acceptance rating which the USEPacks received in previous projects. To measure the acceptance of each USEPack after deployment, the approach proposes to use existing theories of technology acceptance. Based on these theories the perceived usefulness and ease of use of each USEPack deployed in a project is assessed by the project team to measure the acceptance of the USEPack. The approach provides concepts which exploit the collected data to adapt the context profiles of the deployed USEPacks. After a number of iterations the context profiles of USEPacks reflect the factors which contributed to the acceptance or rejection of a USEPack by project teams. This knowledge can be used in future projects to guide the selection of appropriate methods and to improve the acceptance of usability engineering methods by project teams. The concepts of the adoption-centric usability engineering approach are implemented in a support system which aims at supporting the introduction, evaluation and improvement of usability engineering methods in software development organizations. The system is called ProUSE. The main components of the system allow project teams to configure project specific method kits, assess the acceptance of the deployed USEPacks as well as to build and maintain the pool of USEPacks. ProUSE fuses and exploits the data gathered in the acceptance assessments to optimize the project specific selection of appropriate usability engineering methods. In this work an evaluation of the developed framework with potential users of the adoption-centric usability engineering approach is performed. 44 project team members of five software development organizations participated in the evaluation. The evaluation shows that project team members understand the objectives of the approach and can transfer the concepts to their processes. Moreover the evaluation shows that the approach as it is embodied in the support environment ProUSE is perceived as useful and easy to use by project teams. The evaluation indicates that the adoption-centric usability engineering approach has the potential to improve the introduction and establishment of usability engineering methods in software development processes.

[1]  Ian Sommerville,et al.  Software engineering (5th ed.) , 1995 .

[2]  H. Rex Hartson,et al.  Developing user interfaces: ensuring usability through product & process , 1993 .

[3]  Andrew Dillon,et al.  The trouble with computers: Usefulness, usability, and productivity , 1996 .

[4]  Fabio Paternò,et al.  Towards a UML for Interactive Systems , 2001, EHCI.

[5]  James O. Coplien,et al.  Pattern languages of program design , 1995 .

[6]  Thomas P. Moran,et al.  Questions, Options, and Criteria: Elements of Design Space Analysis , 1991, Hum. Comput. Interact..

[7]  Miles Macleod,et al.  The Development of DRUM: A Software Tool for Video-assisted Usability Evaluation , 1998 .

[8]  Thomas K. Landauer,et al.  Trouble with Computers: Usefulness, Usability, and Productivity , 1996 .

[9]  Robert T. Hughes,et al.  Expert judgement as an estimating method , 1996, Inf. Softw. Technol..

[10]  Peter Gorny,et al.  EXPOSE. HCI-counseling for user interface design , 1995, INTERACT.

[11]  G. Huber Organizational Learning: The Contributing Processes and the Literatures , 1991 .

[12]  Robert L. Glass,et al.  A structure-based critique of contemporary computing research , 1995, J. Syst. Softw..

[13]  Ahmed Seffah,et al.  Comparing Use Cases and Task Analysis: A Concrete Example , 1999, ECOOP Workshops.

[14]  Ian I. Mitroff,et al.  Enhancing Organizational Research Utilization: The Role of Decision Makers' Assumptions , 1984 .

[15]  Harald Reiterer,et al.  Use and Reuse of HCI Knowledge in the Software Development Lifecycle: Existing Approaches and What Developers Thinks , 2002, Usability.

[16]  David R. Schwartz,et al.  The use of guidelines in interface design , 1991, CHI.

[17]  Fred D. Davis A technology acceptance model for empirically testing new end-user information systems : theory and results , 1985 .

[18]  Eduard Metzker,et al.  REUSE: Computer Aided Improvement of Human- Centered Design Processes , 2001, Mensch & Computer.

[19]  W. W. Royce,et al.  Managing the development of large software systems , 1970 .

[20]  Harald Reiterer,et al.  Software evaluation using the 9241 evaluator , 1997, Behav. Inf. Technol..

[21]  Anja Becker,et al.  Welche Unterstützung wünschen Softwareentwickler beim Entwurf von Bedienoberflächen? , 2001, Mensch & Computer.

[22]  Shari Lawrence Pfleeger,et al.  Understanding and improving technology transfer in software engineering , 1999, J. Syst. Softw..

[23]  Anke Dittmar,et al.  Relations between Use Cases and Task Analysis , 1999, ECOOP Workshops.

[24]  John Edgar Tidwell,et al.  Common ground: a pattern language for human-computer interface design , 1997 .

[25]  Frank Land,et al.  Choosing Appropriate Information Systems Research Methodologies , 2002 .

[26]  Blair H. Sheppard,et al.  The Theory of Reasoned Action: A Meta-Analysis of Past Research with Recommendations for Modifications and Future Research , 1988 .

[27]  T.M. Duffy,et al.  Scenario-Based Design: Envisioning Work and Technology in System Development [Book Review] , 1996, IEEE Transactions on Professional Communication.

[28]  W. Gibbs Taking Computers to Task , 1997 .

[29]  John Long,et al.  Towards a conception for an engineering discipline of human factors. , 1989, Ergonomics.

[30]  John M. Carroll,et al.  Making use is more than a matter of task analysis , 2002, Interact. Comput..

[31]  Randolph G. Bias,et al.  Cost-justifying usability , 2005 .

[32]  Anne Miller Integrating human factors in customer support systems development using a multi-level organisational approach , 1996, CHI '96.

[33]  Tridas Mukhopadhyay,et al.  Determinants of success in software measurement programs: initial results , 1999, Proceedings Sixth International Software Metrics Symposium (Cat. No.PR00403).

[34]  Brad A. Myers,et al.  User interface software tools , 1995, TCHI.

[35]  A. Adam Whatever happened to information systems ethics? Caught between the devil and the deep blue sea , 2004 .

[36]  Adam A. Porter,et al.  Empirical studies of software engineering: a roadmap , 2000, ICSE '00.

[37]  Forrest Shull,et al.  Building Knowledge through Families of Experiments , 1999, IEEE Trans. Software Eng..

[38]  Elizabeth A. Buie,et al.  User interface guidelines and standards: progress, issues, and prospects , 1999, Interact. Comput..

[39]  Philippe Kruchten Use-Case Storyboards in the Rational Unified Process , 1999, ECOOP Workshops.

[40]  Harald Reiterer,et al.  Evidence-Based Usability Engineering , 2002, CADUI.

[41]  KitchenhamBarbara Ann,et al.  Evaluating software engineering methods and tools , 1998 .

[42]  Alistair Cockburn,et al.  Structuring Use Cases with Goals , 2000 .

[43]  Larry L. Constantine,et al.  Software for Use - A Practical Guide to the Models and Methods of Usage-Centered Design , 1999 .

[44]  Christophe Kolski,et al.  An HCI-Enriched Model for Supporting Human-Machine Systems Design and Evaluation , 1998 .

[45]  John Karat,et al.  Practical education for improving software usability , 1995, CHI '95.

[46]  Barry G. Silverman,et al.  Software Cost and Productivity Improvements: An Analogical View , 1985, Computer.

[47]  ALISTAIR SUTCLIFFE,et al.  On the effective use and reuse of HCI knowledge , 2000, TCHI.

[48]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[49]  Kee Yong Lim,et al.  The MUSE Method for Usability Engineering , 1997, INTERACT.

[50]  A. Seffah,et al.  Towards a Systematic and Empirical Validation of HCI Knowledge Captured as Patterns , 2003 .

[51]  Dennis F. Galletta,et al.  Extending the technology acceptance model to account for social influence: theoretical bases and empirical validation , 1999, Proceedings of the 32nd Annual Hawaii International Conference on Systems Sciences. 1999. HICSS-32. Abstracts and CD-ROM of Full Papers.

[52]  Hartmut Wandke,et al.  Developing a Process Model for the Design of Assistance Components in Information Appliances , 2001 .

[53]  Gardner Murphy,et al.  Experimental social psychology. , 1934 .

[54]  H. D. Rombach,et al.  THE EXPERIENCE FACTORY , 1999 .

[55]  Izak Benbasat,et al.  Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation , 1991, Inf. Syst. Res..

[56]  Scott Henninger,et al.  A methodology and tools for applying context-specific usability guidelines to interface design , 2000, Interact. Comput..

[57]  Dominique L. Scapin,et al.  Organizing human factors knowledge for the evaluation and design of interfaces , 1990, Int. J. Hum. Comput. Interact..

[58]  Karel Vredenburg,et al.  User-centered design methods in practice: a survey of the state of the art , 2001, CASCON.

[59]  Deborah J. Mayhew,et al.  The usability engineering lifecycle , 1999, CHI Extended Abstracts.

[60]  R. E. Kurt Stirewalt,et al.  The MASTERMIND User Interface Generation Project , 1996 .

[61]  Hans van Vliet,et al.  Measurements Should Generate Value, Rather than Data , 1999 .

[62]  TR,et al.  Information technology — Software process assessment — Part 2 : A reference model for processes and process capability , 1998 .

[63]  Salvatore T. March,et al.  Design and natural science research on information technology , 1995, Decis. Support Syst..

[64]  Norman W. Paton,et al.  A UML-based design environment for interactive applications , 2001, Proceedings Second International Workshop on User Interfaces in Data Intensive Systems. UIDIS 2001.

[65]  Peter Constable,et al.  DSDM: Dynamic Systems Development Method: The Method in Practice , 1997 .

[66]  P. Kidwell,et al.  The trouble with computers: Usefulness, usability and productivity , 1996, IEEE Annals of the History of Computing.

[67]  Eliot R. Smith,et al.  Research methods in social relations , 1962 .

[68]  Patricia A. Billingsley Starting from scratch: building a usability program at Union Pacific Railroad , 1995, INTR.

[69]  Donald A. Norman,et al.  The invisible computer , 1998 .

[70]  Donald A. Schön,et al.  Organizational Learning: A Theory Of Action Perspective , 1978 .

[71]  Dan Diaper,et al.  Scenarios and task analysis , 2002, Interact. Comput..

[72]  Marvin V. Zelkowitz,et al.  EXPERIMENTAL MODELS FOR VALIDATING COMPUTER TECHNOLOGY , 2001 .

[73]  James L. Bossert,et al.  Quality Function Deployment , 2021, Quality Function Deployment.

[74]  David Maulsby,et al.  MOBI-D: a model-based development environment for user-centered design , 1997, CHI Extended Abstracts.

[75]  Jean Vanderdonckt,et al.  Development milestones towards a tool for working with guidelines , 1999, Interact. Comput..

[76]  Panos Markopoulos,et al.  UML as a representation for Interaction Design , 2000 .

[77]  I. Ajzen,et al.  Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research , 1977 .

[78]  Frank M. Shipman,et al.  Formality Considered Harmful: Experiences, Emerging Themes, and Directions on the Use of Formal Representations in Interactive Systems , 1999, Computer Supported Cooperative Work (CSCW).

[79]  Ahmed Seffah,et al.  The obstacles and myths of usability and software engineering , 2004, CACM.

[80]  C.-M. Karat Usability engineering in dollars and cents , 1993, IEEE Software.

[81]  Sally Fincher,et al.  Patterns for HCI and Cognitive Dimensions: Two Halves of the Same Story? , 2002, PPIG.

[82]  Eduard Metzker,et al.  An Interdisciplinary Approach for Successfully Integrating Human-Centered Design Methods into Development Processes Practiced by Industrial Software Development Organizations , 2001, EHCI.

[83]  D. Garvin Building a learning organization. , 1993, Harvard business review.

[84]  Watts S. Humphrey,et al.  Managing the software process , 1989, The SEI series in software engineering.

[85]  John Artim,et al.  Incorporating work, process and task analysis into commercial and industrial object-oriented systems development , 1998, SGCH.

[86]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[87]  Matthias Jarke Scenarios for modeling , 1999, CACM.

[88]  P. Jones Making Decisions , 1971, Nature.

[89]  Evangelos Triantaphyllou,et al.  Multi-criteria Decision Making Methods: A Comparative Study , 2000 .

[90]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[91]  Constantine Stephanidis,et al.  Integrated support for working with guidelines: the Sherlock guideline management system , 2000, Interact. Comput..

[92]  J. F. Kelley,et al.  An iterative design methodology for user-friendly natural language office information applications , 1984, TOIS.

[93]  A. R. Crathorne,et al.  Economic Control of Quality of Manufactured Product. , 1933 .

[94]  Karan Harbison,et al.  User-Centered Requirements: The Scenario-Based Engineering Process , 1997 .

[95]  Karen Holtzblatt,et al.  Contextual design , 1997, INTR.

[96]  Mary Beth Rosson,et al.  Usability Engineering: Scenario-based Development of Human-Computer Interaction , 2001 .

[97]  Eduard Metzker,et al.  An experimental process metrics support environment and a cross-organizational study on its acceptance by practitioners , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[98]  Sidney L. Smith,et al.  Guidelines for Designing User Interface Software , 1986 .

[99]  Peter Forbrig,et al.  Task and Object-Oriented Development of Interactive Systems - How many models are necessary? , 1999, DSV-IS.

[100]  Frank Houdek Empirisch basierte Qualitätsverbesserung: systematischer Einsatz externer Experimente im Software Engineering , 1999 .

[101]  Ivar Jacobson,et al.  Object-oriented software engineering - a use case driven approach , 1993, TOOLS.

[102]  George M. Donahue Usability and the Bottom Line , 2001, IEEE Softw..

[103]  Harald Reiterer,et al.  Integrating usability engineering methods into existing software development processes via evidence-based usability engineering , 2004 .

[104]  Gerrit C. van der Veer,et al.  Breaking Down Usability , 1999, INTERACT.

[105]  Torgeir Dingsøyr,et al.  Postmortem: Never Leave a Project without It , 2002, IEEE Softw..

[106]  Stephanie Rosenbaum,et al.  Usability in Practice: user experience lifecycle - evolution and revolution , 2002, CHI Extended Abstracts.

[107]  Frank Lonczewski,et al.  The FUSE-System: an Integrated User Interface Design Environment , 1996, CADUI.

[108]  Kenneth A. Kozar,et al.  Adopting Systems Development Methods: An Exploratory Study , 1989, J. Manag. Inf. Syst..

[109]  David Benyon,et al.  Scenarios and the HCI-SE design problem , 2002, Interact. Comput..

[110]  Leonard J. Bass,et al.  Bridging the gaps between software engineering and human-computer interaction , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[111]  Brian Henderson-Sellers,et al.  The object-oriented systems life cycle , 1990, CACM.

[112]  Mary Beth Rosson,et al.  Integrating development of task and object models , 1999, CACM.

[113]  Tom DeMarco,et al.  Peopleware: Productive Projects and Teams , 1987 .

[114]  Roger S. Pressman,et al.  Software Engineering: A Practitioner's Approach , 1982 .

[115]  Fabio Paternò Commentary on ‘scenarios and task analysis’ by Dan Diaper , 2002 .

[116]  Susan Weinschenk,et al.  Guidelines for Enterprise-Wide Gui Design , 1995 .

[117]  Rick Spencer,et al.  The streamlined cognitive walkthrough method, working around social constraints encountered in a software development company , 2000, CHI.

[118]  Simon Buckingham Shum,et al.  Analyzing the Usability of a Design Rationale Notation , 1996 .

[119]  Xris Faulkner,et al.  Usability Engineering , 2000 .

[120]  David Lei,et al.  Management practices in learning organizations , 1992 .

[121]  Peter F. Conklin,et al.  Bringing usability effectively into product development , 1995 .

[122]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[123]  Pertti Järvinen,et al.  Research Questions Guiding Selection of an Appropriate Research Method , 2000, ECIS.

[124]  A. Bonaert Introduction to the theory of Fuzzy subsets , 1977, Proceedings of the IEEE.

[125]  Barry W. Boehm,et al.  A spiral model of software development and enhancement , 1986, Computer.

[126]  Fred D. Davis,et al.  User Acceptance of Computer Technology: A Comparison of Two Theoretical Models , 1989 .

[127]  Juliet Sutherland,et al.  SCRUM: An extension pattern language for hyper productive software development , 2000 .

[128]  Oliver Laitenberger,et al.  Evaluating the usefulness and the ease of use of a Web-based inspection data collection tool , 1998, Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262).

[129]  Andy Smith,et al.  User Centred Design: The Application of the LUCID Interface Design Method , 1997, HCI.

[130]  Daniel Robey,et al.  Research Commentary: Diversity in Information Systems Research: Threat, Promise, and Responsibility , 1996, Inf. Syst. Res..

[131]  Leonard J. Bass,et al.  Linking usability to software architecture patterns through general scenarios , 2003, J. Syst. Softw..

[132]  Natalia Juristo Juzgado,et al.  Basics of Software Engineering Experimentation , 2010, Springer US.

[133]  Jakob Nielsen,et al.  Information Appliances and Beyond , 2000 .

[134]  Jakob Nielsen,et al.  Heuristic Evaluation of Prototypes (individual) , 2022 .

[135]  I. Ajzen,et al.  Prediction of goal directed behaviour: Attitudes, intentions and perceived behavioural control , 1986 .

[136]  A. R. Ilersic,et al.  Research methods in social relations , 1961 .

[137]  Michael A. Jackson,et al.  Software requirements and specifications - a lexicon of practice, principles and prejudices , 1995 .