Do Security Reports Meet Usability?: Lessons Learned from Using Actionable Mitigations for Patching TLS Misconfigurations

Several automated tools have been proposed to detect vulnerabilities. These tools are mainly evaluated in terms of their accuracy in detecting vulnerabilities, but the evaluation of their usability is a commonly neglected topic. Usability of automated security tools is particularly crucial when dealing with problems of cryptographic protocols for which even small—apparently insignificant—changes in their configuration can result in vulnerabilities that, if exploited, pave the way to attacks with dramatic consequences for the confidentiality and integrity of exchanged messages. This becomes even more acute when considering such ubiquitous protocols as the one for Transport Layer Security (TLS for short). In this paper, we present the design and the lessons learned of a user study, meant to compare two different approaches when reporting misconfigurations. Results reveal that including contextualized actionable mitigations in security reports significantly impact the accuracy and the time needed to patch TLS vulnerabilities. Along with the lessons learned, we share the experimental material that can be used during cybersecurity labs to let students configure and patch TLS first-hand.

[1]  Michael Backes,et al.  You Get Where You're Looking for: The Impact of Information Sources on Code Security , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[2]  K. Goulden,et al.  Effect Sizes for Research: A Broad Practical Approach , 2006 .

[3]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[4]  Sabrina Marczak,et al.  Model comprehension for security risk assessment: an empirical comparison of tabular vs. graphical representations , 2017, Empirical Software Engineering.

[5]  Marco Torchiano,et al.  Assessment of Source Code Obfuscation Techniques , 2016, 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM).

[6]  Bodo Möller,et al.  This POODLE Bites: Exploiting The SSL 3.0 Fallback , 2014 .

[7]  Eric Rescorla,et al.  The Transport Layer Security (TLS) Protocol Version 1.1 , 2006, RFC.

[8]  Fabio Massacci,et al.  An Experimental Comparison of Two Risk-Based Security Methods , 2013, 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement.

[9]  Donald E. Myers,et al.  Linear and Generalized Linear Mixed Models and Their Applications , 2008, Technometrics.

[10]  Katharina Krombholz,et al.  "If HTTPS Were Secure, I Wouldn't Need 2FA" - End User and Administrator Mental Models of HTTPS , 2019, 2019 IEEE Symposium on Security and Privacy (SP).

[11]  Scott Hollenbeck,et al.  Transport Layer Security Protocol Compression Methods , 2004, RFC.

[12]  Matthew Smith,et al.  Why Do Developers Get Password Storage Wrong?: A Qualitative Usability Study , 2017, CCS.

[13]  Matthew Smith,et al.  "If you want, I can store the encrypted password": A Password-Storage Field Study with Freelance Developers , 2019, CHI.

[14]  Edgar R. Weippl,et al.  "I Have No Idea What I'm Doing" - On the Usability of Deploying HTTPS , 2017, USENIX Security Symposium.

[15]  Frank Li,et al.  Keepers of the Machines: Examining How System Administrators Manage Software Updates For Multiple Machines , 2019, SOUPS @ USENIX Security Symposium.

[16]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[17]  Eric Rescorla,et al.  The Transport Layer Security (TLS) Protocol Version 1.2 , 2008, RFC.

[18]  Claes Wohlin,et al.  Using students as subjects - an empirical evaluation , 2008, ESEM '08.

[19]  Karthikeyan Bhargavan,et al.  On the Practical (In-)Security of 64-bit Block Ciphers: Collision Attacks on HTTP over TLS and OpenVPN , 2016, CCS.

[20]  Natalia Juristo Juzgado,et al.  Are Students Representatives of Professionals in Software Engineering Experiments? , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[21]  Fabio Massacci,et al.  Measuring the accuracy of software vulnerability assessments: experiments with students and professionals , 2020, Empirical Software Engineering.

[22]  Stephen Farrell,et al.  Deprecating TLSv1.0 and TLSv1.1 , 2020 .

[23]  Michelle Cartwright.  Book Review: Experimentation in Software Engineering: An Introduction. By Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell and Anders Wesslén. Kluwer Academic Publishers, 1999, ISBN 0-7923-8682-5 , 2001, Softw. Test. Verification Reliab..

[24]  P. Lachenbruch.  Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[25]  Marco Torchiano,et al.  Empirical assessment of the effort needed to attack programs protected with client/server code splitting , 2019, Empirical Software Engineering.

[26]  Riccardo Scandariato,et al.  Static Analysis and Penetration Testing from the Perspective of Maintenance Teams , 2016, ESEM.

[27]  Fabio Massacci,et al.  Estimating the Assessment Difficulty of CVSS Environmental Metrics: An Experiment , 2017, FDSE.

[28]  Mariano Ceccato,et al.  Do Automatically Generated Test Cases Make Debugging Easier? An Experimental Assessment of Debugging Effectiveness and Efficiency , 2015, ACM Trans. Softw. Eng. Methodol..

[29]  S. Greven,et al.  A unifying approach to the estimation of the conditional Akaike information in generalized linear mixed models , 2014 .

[30]  Healthy Aging Lab.  I have no idea what I'm doing, but I like guys. , 2018 .

[31]  Eric R. Ziegel,et al.  Probability and Statistics for Engineering and the Sciences , 2004, Technometrics.

[32]  Wouter Joosen,et al.  Static analysis versus penetration testing: A controlled experiment , 2013, 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE).

[33]  Felix C. Freiling,et al.  Programming Experience Might Not Help in Comprehending Obfuscated Source Code Efficiently , 2018, SOUPS @ USENIX Security Symposium.

[34]  The significance fallacy in inferential statistics , 2015, BMC Research Notes.

[35]  Vitaly Shmatikov,et al.  The most dangerous code in the world: validating SSL certificates in non-browser software , 2012, CCS.

[36]  A. Strauss,et al.  The Discovery of Grounded Theory , 1967 .

[37]  Pamela Jordan.  Basics of qualitative research: Grounded theory procedures and techniques , 1994 .

[38]  Marco Torchiano,et al.  The effectiveness of source code obfuscation: An experimental assessment , 2009, 2009 IEEE 17th International Conference on Program Comprehension.

[39]  Dan S. Wallach,et al.  On the Usability of HTTPS Deployment , 2019, CHI.

[40]  Marco Torchiano,et al.  A family of experiments to assess the effectiveness and efficiency of source code obfuscation techniques , 2013, Empirical Software Engineering.

[41]  Mario Cortina-Borja,et al.  Handbook of Parametric and Nonparametric Statistical Procedures, 5th edn , 2012 .

[42]  Peter Saint-Andre,et al.  Summarizing Known Attacks on Transport Layer Security (TLS) and Datagram TLS (DTLS) , 2015, RFC.

[43]  Katharina Krombholz,et al.  Security, Availability, and Multiple Information Sources: Exploring Update Behavior of System Administrators , 2020, SOUPS @ USENIX Security Symposium.