The Security Blanket of the Chat World: An Analytic Evaluation and a User Study of Telegram

The computer security community has advocated widespread adoption of secure communication tools to protect personal privacy. Several popular communication tools have adopted end-to-end encryption (e.g., WhatsApp, iMessage), or promoted security features as selling points (e.g., Telegram, Signal). However, previous studies have shown that users may not understand the security features of the tools they are using, and may not be using them correctly. In this paper, we present a study of Telegram using two complementary methods: (1) a labbased user study (11 novices and 11 Telegram users), and (2) a hybrid analytical approach combining cognitive walk-through and heuristic evaluation to analyse Telegram’s user interface. Participants who use Telegram feel secure because they feel they are using a secure tool, but in reality Telegram offers limited security benefits to most of its users. Most participants develop a habit of using the less secure default chat mode at all times. We also uncover several user interface design issues that impact security, including technical jargon, inconsistent use of terminology, and making some security features clear and others not. For instance, use of the end-to-end-encrypted Secret Chat mode requires both the sender and recipient be online at the same time, and Secret Chat does not support group conversations.

[1]  Ian Goldberg,et al.  Improved user authentication in off-the-record messaging , 2007, WPES '07.

[2]  Jakob Jakobsen,et al.  On the CCA (in)Security of MTProto , 2015, SPSM@CCS.

[3]  Paul Dourish,et al.  Security in the wild: user strategies for managing security as an everyday, practical problem , 2004, Personal and Ubiquitous Computing.

[4]  Cathleen Wharton,et al.  The cognitive walkthrough method: a practitioner's guide , 1994 .

[5]  Umit Topaloglu,et al.  Off-the-Record Instant Messaging for Group Conversation , 2007, 2007 IEEE International Conference on Information Reuse and Integration.

[6]  Amit A. Levy,et al.  Vanish: Increasing Data Privacy with Self-Destructing Data , 2009, USENIX Security Symposium.

[7]  Ian Goldberg,et al.  A user study of off-the-record messaging , 2008, SOUPS '08.

[8]  Ian Goldberg,et al.  SoK: Secure Messaging , 2015, 2015 IEEE Symposium on Security and Privacy.

[9]  Andrew Sears,et al.  Heuristic Walkthroughs: Finding the Problems Without the Noise , 1997, Int. J. Hum. Comput. Interact..

[10]  Doowon Kim,et al.  An Inconvenient Trust: User Attitudes toward Security and Usability Tradeoffs for Key-Directory Encryption Systems , 2016, SOUPS.

[11]  Mary Beth Rosson,et al.  Paradox of the active user , 1987 .

[12]  Kat Krol,et al.  Better the Devil You Know: A User Study of Two CAPTCHAs and a Possible Replacement , 2016 .

[13]  Mark Ryan,et al.  Enhanced Certificate Transparency and End-to-End Encrypted Mail , 2014, NDSS.

[14]  Peter Gutmann,et al.  PKI: It's Not Dead, Just Resting , 2002, Computer.

[15]  Hong Liu,et al.  Improved group off-the-record messaging , 2013, WPES.

[16]  Whitfield Diffie,et al.  New Directions in Cryptography , 1976, IEEE Trans. Inf. Theory.

[17]  Srdjan Capkun,et al.  SoK: Secure Data Deletion , 2013, 2013 IEEE Symposium on Security and Privacy.

[18]  Ian Goldberg,et al.  Multi-party off-the-record messaging , 2009, CCS.

[19]  Michael E. Atwood,et al.  What is gained and lost when using methods other than empirical testing , 1992, CHI '92.

[20]  Gerhard Fischer,et al.  Supporting learning on demand with design environments , 1991 .

[21]  Clayton Lewis,et al.  TASK-CENTERED USER INTERFACE DESIGN A Practical Introduction , 2006 .

[22]  Radia Perlman,et al.  The ephemerizer: making data disappear , 2005 .

[23]  Cathleen Wharton,et al.  Testing a walkthrough methodology for theory-based design of walk-up-and-use interfaces , 1990, CHI '90.

[24]  Philip R. Zimmermann,et al.  The official PGP user's guide , 1996 .

[25]  白石 善明,et al.  "Confused Johnny: When Automatic Encryption Leads to Confusion and Mistakes"の紹介 , 2013 .

[26]  Edward W. Felten,et al.  Secrecy, flagging, and paranoia: adoption criteria in encrypted email , 2006, CHI.

[27]  Jakob Nielsen,et al.  Heuristic evaluation of user interfaces , 1990, CHI '90.

[28]  David G. Novick,et al.  Usability inspection methods after 15 years of research and practice , 2007, SIGDOC '07.

[29]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[30]  Melanie Volkamer,et al.  Why Doesn't Jane Protect Her Privacy? , 2014, Privacy Enhancing Technologies.

[31]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[32]  Laura A. Dabbish,et al.  The Effect of Social Influence on Security Sensitivity , 2014, SOUPS.

[33]  Kat Krol,et al.  Appropriation of security technologies in the workplace , 2015 .

[34]  Ann Cavoukian,et al.  Privacy by Design [Leading Edge] , 2012, IEEE Technol. Soc. Mag..

[35]  M. Angela Sasse,et al.  Obstacles to the Adoption of Secure Communication Tools , 2017, 2017 IEEE Symposium on Security and Privacy (SP).

[36]  Robin Jeffries,et al.  User interface evaluation in the real world: a comparison of four techniques , 1991, CHI.

[37]  Cathleen Wharton,et al.  Cognitive Walkthroughs: A Method for Theory-Based Evaluation of User Interfaces , 1992, Int. J. Man Mach. Stud..

[38]  Daniel Zappala,et al.  "We're on the Same Page": A Usability Study of Secure Email Using Pairs of Novice Users , 2015, CHI.

[39]  J. Doug Tygar,et al.  Why Johnny Can't Encrypt: A Usability Evaluation of PGP 5.0 , 1999, USENIX Security Symposium.

[40]  Kat Krol,et al.  Towards Robust Experimental Design for User Studies in Security and Privacy , 2016 .

[41]  Brian Shackel,et al.  Human Factors and Usability. , 1990, Studies in health technology and informatics.

[42]  James T. Miller,et al.  An Empirical Evaluation of the System Usability Scale , 2008, Int. J. Hum. Comput. Interact..

[43]  Martin Ortlieb,et al.  Expert and Non-Expert Attitudes towards (Secure) Instant Messaging , 2016, SOUPS.

[44]  Thomas S. Tullis,et al.  A Comparison of Questionnaires for Assessing Website Usability , 2004 .

[45]  Amir Herzberg,et al.  Can Johnny finally encrypt?: evaluating E2E-encryption in popular IM applications , 2016, STAST.

[46]  Nikita Borisov,et al.  Off-the-record communication, or, why not to use PGP , 2004, WPES '04.

[47]  Rob Miller,et al.  Johnny 2: a user test of key continuity management with S/MIME and Outlook Express , 2005, SOUPS '05.

[48]  V. Braun,et al.  Using thematic analysis in psychology , 2006 .

[49]  Tadayoshi Kohno,et al.  Sex, Lies, or Kittens? Investigating the Use of Snapchat's Self-Destructing Messages , 2014, Financial Cryptography.

[50]  Clare-Marie Karat,et al.  Comparison of empirical testing and walkthrough methods in user interface evaluation , 1992, CHI.

[51]  Michael J. Freedman,et al.  CONIKS: Bringing Key Transparency to End Users , 2015, USENIX Security Symposium.