Heuristics for evaluating IT security management tools

The usability of IT security management (ITSM) tools is hard to evaluate by regular methods, making heuristic evaluation attractive. However, ITSM occurs within a complex and collaborative context that involves diverse stakeholders; this makes standard usability heuristics difficult to apply. We propose a set of ITSM usability heuristics that are based on activity theory and supported by prior research. We performed a study to compare the use of the ITSM heuristics to Nielsen's heuristics for the evaluation of a commercial identity management system. Our preliminary results show that our new ITSM heuristics performed well in finding usability problems. However, we need to perform the study with more participants and perform more detailed analysis to precisely show the differences in applying the ITSM heuristics as compared to Nielsen's heuristics.

[1]  Dov Te'eni,et al.  Human-Computer Interaction: Developing Effective Organizational Information Systems , 2006 .

[2]  Carl Gutwin,et al.  Adapting the Locales Framework for Heuristic Evaluation of Groupware , 2000, Australas. J. Inf. Syst..

[3]  Eben M. Haber,et al.  Design guidelines for system administration tools developed through ethnographic field studies , 2007, CHIMIT '07.

[4]  Geraldine Fitzpatrick,et al.  rk: Exploring founda , 1996 .

[5]  Carl Gutwin,et al.  Heuristic Evaluation of Groupware Based on the Mechanics of Collaboration , 2001, EHCI.

[6]  Kirstie Hawkey,et al.  An integrated view of human, organizational, and technological challenges of IT security management , 2009, Inf. Manag. Comput. Secur..

[7]  K. Kuutti.  Activity theory as a potential framework for human-computer interaction research , 1995 .

[8]  Alistair G. Sutcliffe,et al.  Heuristic evaluation of virtual reality applications , 2004, Interact. Comput..

[9]  Thomas P. Moran,et al.  Commentary on "Damaged Merchandise?" , 1998, Hum. Comput. Interact..

[10]  Konstantin Beznosov,et al.  Towards understanding IT security professionals and their tools , 2007, SOUPS '07.

[11]  P. Sarbanes,et al.  Sarbanes-Oxley Act of 2002 , 2002 .

[12]  Keith Duncan,et al.  Cognitive Engineering , 2017, Encyclopedia of GIS.

[13]  William Yurcik,et al.  Command line or pretty lines?: comparing textual and visual interfaces for intrusion detection , 2007, CHI.

[14]  Michael J. Muller,et al.  Validating an extension to participatory heuristic evaluation: quality of work and quality of work life , 1995, CHI 95 Conference Companion.

[15]  Y. Engeström,et al.  Activity theory and individual and social transformation. , 1999 .

[16]  Mary Beth Rosson,et al.  Notification and awareness: synchronizing task-oriented collaborative activity , 2003, Int. J. Hum. Comput. Stud..

[17]  David Zager,et al.  Collaboration as an Activity Coordinating with Pseudo-Collective Objects , 2002, Computer Supported Cooperative Work (CSCW).

[18]  Bonnie A. Nardi,et al.  NetWORKers and their Activity in Intensional Networks , 2002, Computer Supported Cooperative Work (CSCW).

[19]  Carl Gutwin,et al.  Empirical development of a heuristic evaluation methodology for shared workspace groupware , 2002, CSCW '02.

[20]  A.,et al.  Cognitive Engineering , 2008, Encyclopedia of GIS.

[21]  Kasia Muldner,et al.  Identifying Differences between Security and other IT Professionals: a Qualitative Analysis , 2008, HAISA.

[22]  Carl Gutwin,et al.  The mechanics of collaboration: developing low cost usability evaluation methods for shared workspaces , 2000, Proceedings IEEE 9th International Workshops on Enabling Technologies: Infrastructure for Collaborative Enterprises (WET ICE 2000).

[23]  Mary Beth Rosson,et al.  Usability Engineering: Scenario-based Development of Human-Computer Interaction , 2001 .

[24]  Karel Vredenburg,et al.  A survey of user-centered design practice , 2002, CHI.

[25]  Paul Dourish,et al.  An approach to usable security based on event monitoring and visualization , 2002, NSPW '02.

[26]  Mary Beth Rosson,et al.  Evaluating computer-supported cooperative work: models and frameworks , 2004, CSCW.

[27]  Donald A. Norman,et al.  Cognitive artifacts , 1991 .

[28]  James D. Hollan,et al.  Distributed cognition: toward a new foundation for human-computer interaction research , 2000, TCHI.

[29]  Wendy A. Kellogg,et al.  Social translucence: an approach to designing systems that support social processes , 2000, TCHI.

[30]  Kirstie Hawkey,et al.  Guidelines for designing IT security management tools , 2008, CHiMiT '08.

[31]  Pauline Ratnasingam,et al.  A knowledge architecture for IT security , 2007, CACM.

[32]  Kim J. Vicente,et al.  HCI in the global knowledge-based economy: designing to support worker adaptation , 2000, TCHI.

[33]  Mary S. Schaeffer,et al.  Sarbanes-Oxley Act of 2002 , 2012 .

[34]  Wayne G. Lutters,et al.  I know my network: collaboration and expertise in intrusion detection , 2004, CSCW.

[35]  Kirstie Hawkey,et al.  Security practitioners in context: their activities and interactions , 2008, Int. J. Hum. Comput. Stud..

[36]  Jakob Nielsen,et al.  A mathematical model of the finding of usability problems , 1993, INTERCHI.

[37]  Lujo Bauer,et al.  Real life challenges in access-control management , 2009, CHI.

[38]  A.T. Zhou,et al.  Improving intrusion detection systems through heuristic evaluation , 2004, Canadian Conference on Electrical and Computer Engineering 2004 (IEEE Cat. No.04CH37513).

[39]  Kasia Muldner,et al.  Toward understanding distributed cognition in IT security management: the role of cues and norms , 2011, Cognition, Technology & Work.

[40]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[41]  Jakob Nielsen,et al.  How to Conduct a Heuristic Evaluation , 2006 .

[42]  James D. Hollan,et al.  Post-cognitivist HCI: second-wave theories , 2003, CHI Extended Abstracts.

[43]  Vimla L. Patel,et al.  Using usability heuristics to evaluate patient safety of medical devices , 2003, J. Biomed. Informatics.

[44]  B. Nardi.  Context and consciousness: activity theory and human-computer interaction , 1995 .

[45]  Nicole F. Velasquez,et al.  Sysadmins and the need for verification information , 2008, CHiMiT '08.

[46]  Ben Shneiderman,et al.  Designing the User Interface: Strategies for Effective Human-Computer Interaction , 1998 .

[47]  B. Beal.  Vendor Analysis: IT security: the product vendor landscape , 2005 .

[48]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[49]  Robin Jeffries,et al.  User interface evaluation in the real world: a comparison of four techniques , 1991, CHI.

[50]  Jakob Nielsen,et al.  Finding usability problems through heuristic evaluation , 1992, CHI.

[51]  Jakob Nielsen,et al.  Usability inspection methods , 1994, CHI 95 Conference Companion.

[52]  Nicole F. Velasquez,et al.  Work practices of system administrators: implications for tool design , 2008, CHiMiT '08.

[53]  Jan Guynes Clark,et al.  Why there aren't more information security research studies , 2004, Inf. Manag..

[54]  Robert C. Williges,et al.  Criteria For Evaluating Usability Evaluation Methods , 2001, Int. J. Hum. Comput. Interact..

[55]  Anind K. Dey,et al.  Heuristic evaluation of ambient displays , 2003, CHI '03.

[56]  Kasia Muldner,et al.  Human, organizational, and technological factors of IT security , 2008, CHI Extended Abstracts.

[57]  Gaëtan Bourmaud,et al.  From computer to instrument system: a developmental perspective , 2003, Interact. Comput..

[58]  Victor Kaptelinin,et al.  Acting with technology: Activity theory and interaction design , 2006, First Monday.

[59]  Yvonne Rogers,et al.  Ghosts in the network: distributed troubleshooting in a shared working environment , 1992, CSCW '92.

[60]  Eser Kandogan,et al.  Distributed Cognition and Joint Activity in Collaborative Problem Solving , 2003 .

[61]  David Pinelle,et al.  Heuristic evaluation for games: usability principles for video game design , 2008, CHI.

[62]  Ben Shneiderman,et al.  Creating creativity: user interfaces for supporting innovation , 2000, TCHI.