Dead on Arrival: Recovering from Fatal Flaws in Email Encryption Tools

Background. Since Whitten and Tygar’s seminal study of PGP 5.0 in 1999, there have been continuing efforts to produce email encryption tools for adoption by a wider user base, where these efforts vary in how well they consider the usability and utility needs of prospective users. Aim. We conducted a study aiming to assess the user experience of two open-source encryption software tools – Enigmail and Mailvelope. Method. We carried out a three-part user study (installation, home use, and debrief) with two groups of users using either Enigmail or Mailvelope. Users had access to help during installation (installation guide and experimenter with domain-specific knowledge), and were set a primary task of organising a mock flash mob using encrypted emails in the course of a week. Results. Participants struggled to install the tools – they would not have been able to complete installation without help. Even with help, setup time was around 40 minutes. Participants using Mailvelope failed to encrypt their initial emails due to usability problems. Participants said they were unlikely to continue using the tools after the study, indicating that their creators must also consider utility. Conclusions. Through our mixed study approach, we conclude that Mailvelope and Enigmail had too many software quality and usability issues to be adopted by mainstream users. Methodologically, the study made us rethink the role of the experimenter as that of a helper assisting novice users with setting up a demanding technology.

[1]  Simson L. Garfinkel,et al.  How to make secure email easier to use , 2005, CHI.

[2]  Kat Krol,et al.  The Security Blanket of the Chat World: An Analytic Evaluation and a User Study of Telegram , 2017 .

[3]  Simson L. Garfinkel.  Enabling Email Confidentiality through the use of Opportunistic Encryption , 2003, DG.O.

[4]  Melanie Volkamer,et al.  Bewertung der GMX/Mailvelope-Ende-zu-Ende-Verschlüsselung , 2016, Datenschutz und Datensicherheit - DuD.

[5]  Simon Parkin,et al.  Combining Qualitative Coding and Sentiment Analysis: Deconstructing Perceptions of Usable Security in Organisations , 2016 .

[6]  Edward W. Felten,et al.  Secrecy, flagging, and paranoia: adoption criteria in encrypted email , 2006, CHI.

[7]  Rick Wash,et al.  Identifying patterns in informal sources of security information , 2015, J. Cybersecur..

[8]  M. Angela Sasse,et al.  Obstacles to the Adoption of Secure Communication Tools , 2017, 2017 IEEE Symposium on Security and Privacy (SP).

[9]  Daniel Zappala,et al.  "We're on the Same Page": A Usability Study of Secure Email Using Pairs of Novice Users , 2015, CHI.

[10]  Rob Miller,et al.  Views, Reactions and Impact of Digitally-Signed Mail in e-Commerce , 2005, Financial Cryptography.

[11]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[12]  Melanie Volkamer,et al.  Why Doesn't Jane Protect Her Privacy? , 2014, Privacy Enhancing Technologies.

[13]  Matt Blaze,et al.  Why (Special Agent) Johnny (Still) Can't Encrypt: A Security Analysis of the APCO Project 25 Two-Way Radio System , 2011, USENIX Security Symposium.

[14]  Jakob Nielsen,et al.  Improving a human-computer dialogue , 1990, CACM.

[15]  Scott Ruoti,et al.  Confused Johnny: when automatic encryption leads to confusion and mistakes , 2013, SOUPS.

[16]  James F. Ryan,et al.  Usable Encryption Enabled by AJAX , 2006, International conference on Networking and Services (ICNS'06).

[17]  W. Keith Edwards,et al.  Computer help at home: methods and motivations for informal technical support , 2009, CHI.

[18]  Rob Miller,et al.  Johnny 2: a user test of key continuity management with S/MIME and Outlook Express , 2005, SOUPS '05.

[19]  Daniel Zappala,et al.  There's Hope for Johnny: Automatic vs. Manual Encryption. , 2015 .

[20]  J. Doug Tygar,et al.  Why Johnny Can't Encrypt: A Usability Evaluation of PGP 5.0 , 1999, USENIX Security Symposium.

[21]  Kat Krol,et al.  Towards Robust Experimental Design for User Studies in Security and Privacy , 2016 .

[22]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[23]  Kat Krol,et al.  Don't work. Can't work? Why it's time to rethink security warnings , 2012, 2012 7th International Conference on Risks and Security of Internet and Systems (CRiSIS).