A Comprehensive Quality Evaluation of Security and Privacy Advice on the Web

End users learn defensive security behaviors from a variety of channels, including a plethora of security advice given in online articles. A great deal of effort is devoted to getting users to follow this advice. Surprisingly then, little is known about the quality of this advice: Is it comprehensible? Is it actionable? Is it effective? To answer these questions, we first conduct a large-scale, user-driven measurement study to identify 374 unique recommended behaviors contained within 1,264 documents of online security and privacy advice. Second, we develop and validate measurement approaches for evaluating the quality – comprehensibility, perceived actionability, and perceived efficacy – of security advice. Third, we deploy these measurement approaches to evaluate the 374 unique pieces of security advice in a user-study with 1,586 users and 41 professional security experts. Our results suggest a crisis of advice prioritization. The majority of advice is perceived by the most users to be at least somewhat actionable, and somewhat comprehensible. Yet, both users and experts struggle to prioritize this advice. For example, experts perceive 89% of the hundreds of studied behaviors as being effective, and identify 118 of them as being among the “top 5” things users should do, leaving end-users on their own to prioritize and take action to protect themselves.

[1]  Elissa M. Redmiles,et al.  A Summary of Survey Methodology Best Practices for Security and Privacy Researchers , 2017 .

[2]  M. Angela Sasse,et al.  The compliance budget: managing security behaviour in organisations , 2009, NSPW '08.

[3]  Terry K Koo,et al.  A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. , 2016, Journal Chiropractic Medicine.

[4]  Sunny Consolvo,et al.  "...No one Can Hack My Mind": Comparing Expert and Non-Expert Security Practices , 2015, SOUPS.

[5]  Rick Wash,et al.  Stories as informal lessons about security , 2012, SOUPS.

[6]  Yehuda Koren,et al.  Matrix Factorization Techniques for Recommender Systems , 2009, Computer.

[7]  Joseph S. Dumas,et al.  Comparison of three one-question, post-task usability questionnaires , 2009, CHI.

[8]  Lorrie Faith Cranor,et al.  Your attention please: designing security-decision UIs to make genuine risks harder to ignore , 2013, SOUPS.

[9]  Elissa M. Redmiles,et al.  I Think They're Trying to Tell Me Something: Advice Sources and Selection for Digital Security , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[10]  Kirstie Hawkey,et al.  On the challenges in usable security lab studies: lessons learned from replicating a study on SSL warnings , 2011, SOUPS.

[11]  Ray A. Perlner,et al.  Electronic Authentication Guideline , 2014 .

[12]  Cormac Herley,et al.  Unfalsifiability of security claims , 2016, Proceedings of the National Academy of Sciences.

[13]  Robert Biddle,et al.  The Role of Instructional Design in Persuasion: A Comics Approach for Improving Cybersecurity , 2016, Int. J. Hum. Comput. Interact..

[14]  Blase Ur,et al.  Can Unicorns Help Users Compare Crypto Key Fingerprints? , 2017, CHI.

[15]  Lorrie Faith Cranor,et al.  A Framework for Reasoning About the Human in the Loop , 2008, UPSEC.

[16]  Lorrie Faith Cranor,et al.  A comparative study of online privacy policies and formats , 2009, Privacy Enhancing Technologies.

[17]  Rick Wash,et al.  Understanding Password Choices: How Frequently Entered Passwords Are Re-used across Websites , 2016, SOUPS.

[18]  Michelle L. Mazurek,et al.  The Effect of Entertainment Media on Mental Models of Computer Security , 2019, SOUPS @ USENIX Security Symposium.

[19]  Lorrie Faith Cranor,et al.  School of phish: a real-world evaluation of anti-phishing training , 2009, SOUPS.

[20]  Elissa M. Redmiles,et al.  Comparing and Developing Tools to Measure the Readability of Domain-Specific Texts , 2019, EMNLP/IJCNLP.

[21]  Sunny Consolvo,et al.  152 Simple Steps to Stay Safe Online: Security Advice for Non-Tech-Savvy Users , 2017, IEEE Security & Privacy.

[22]  E. Seydel,et al.  Protection Motivation Theory , 2022 .

[23]  Wilson L. Taylor,et al.  “Cloze Procedure”: A New Tool for Measuring Readability , 1953 .

[24]  Tudor Dumitras,et al.  Patch Me If You Can: A Study on the Effects of Individual User Behavior on the End-Host Vulnerability State , 2017, PAM.

[25]  L. Cranor,et al.  Nudges for Privacy and Security , 2017, ACM Comput. Surv..

[26]  Matthew Smith,et al.  Replication: No One Can Hack My Mind Revisiting a Study on Expert and Non-Expert Security Practices and Advice , 2019, SOUPS @ USENIX Security Symposium.

[27]  James Nicholson,et al.  Introducing the Cybersurvival Task: Assessing and Addressing Staff Beliefs about Effective Cyber Protection , 2018, SOUPS @ USENIX Security Symposium.

[28]  Lorrie Faith Cranor,et al.  Crying Wolf: An Empirical Study of SSL Warning Effectiveness , 2009, USENIX Security Symposium.

[29]  J. Miller,et al.  Evaluating the Readability of Privacy Policies in Mobile Environments , 2011, Int. J. Mob. Hum. Comput. Interact..

[30]  Mohamed Shehab,et al.  The Effectiveness of Fear Appeals in Increasing Smartphone Locking Behavior among Saudi Arabians , 2018, SOUPS @ USENIX Security Symposium.

[31]  L. Jean Camp,et al.  Risk Communication Design: Video vs. Text , 2012, Privacy Enhancing Technologies.

[32]  Florian Schaub,et al.  You `Might' Be Affected: An Empirical Analysis of Readability and Usability Issues in Data Breach Notifications , 2019, CHI.

[33]  Cristian Danescu-Niculescu-Mizil,et al.  Winning Arguments: Interaction Dynamics and Persuasion Strategies in Good-faith Online Discussions , 2016, WWW.

[34]  Lorrie Faith Cranor,et al.  Anti-Phishing Phil: the design and evaluation of a game that teaches people not to fall for phish , 2007, SOUPS '07.

[35]  Rick Wash,et al.  Too Much Knowledge? Security Beliefs and Protective Behaviors Among United States Internet Users , 2015, SOUPS.

[36]  Martha Larson,et al.  Collaborative Filtering beyond the User-Item Matrix , 2014, ACM Comput. Surv..

[37]  Kelly E. Caine,et al.  Exploring everyday privacy behaviors and misclosures , 2009 .

[38]  Cormac Herley,et al.  So long, and no thanks for the externalities: the rational rejection of security advice by users , 2009, NSPW '09.

[39]  Frank Stajano,et al.  The Quest to Replace Passwords: A Framework for Comparative Evaluation of Web Authentication Schemes , 2012, 2012 IEEE Symposium on Security and Privacy.

[40]  Rick Wash,et al.  Can People Self-Report Security Accurately?: Agreement Between Self-Report and Behavioral Measures , 2017, CHI.

[41]  Sunny Consolvo,et al.  Experimenting at scale with google chrome's SSL warning , 2014, CHI.

[42]  Ani Nenkova,et al.  Revisiting Readability: A Unified Framework for Predicting Text Quality , 2008, EMNLP.

[43]  Matthew Smith,et al.  Towards measuring warning readability , 2012, CCS.

[44]  E. McColl.  Cognitive Interviewing. A Tool for Improving Questionnaire Design , 2006, Quality of Life Research.

[45]  Ani Nenkova,et al.  What Makes Writing Great? First Experiments on Article Quality Prediction in the Science Journalism Domain , 2013, TACL.

[46]  Lorrie Faith Cranor,et al.  Improving Computer Security Dialogs , 2011, INTERACT.

[47]  J. R. Landis,et al.  The measurement of observer agreement for categorical data. , 1977, Biometrics.

[48]  R. Flesch.  A new readability yardstick. , 1948, The Journal of applied psychology.

[49]  Elissa M. Redmiles,et al.  Dancing Pigs or Externalities?: Measuring the Rationality of Security Decisions , 2018, EC.

[50]  John W. Oller,et al.  CLOZE TESTS IN ENGLISH, THAI, AND VIETNAMESE: NATIVE AND NON‐NATIVE PERFORMANCE , 1972 .

[51]  Cormac Herley,et al.  More Is Not the Answer , 2014, IEEE Security & Privacy.

[52]  Markus Jakobsson,et al.  Using Cartoons to Teach Internet Security , 2008, Cryptologia.

[53]  Akira Yamada,et al.  Self-Confidence Trumps Knowledge: A Cross-Cultural Study of Security Behavior , 2017, CHI.

[54]  Elissa M. Redmiles,et al.  How I Learned to be Secure: a Census-Representative Survey of Security Advice Sources and Behavior , 2016, CCS.

[55]  Noah A. Smith,et al.  Automatic factual question generation from text , 2011 .

[56]  Martin Pielot,et al.  Make It Big!: The Effect of Font Size and Line Spacing on Online Readability , 2016, CHI.

[57]  Tadayoshi Kohno,et al.  Control-Alt-Hack: the design and evaluation of a card game for computer security awareness and education , 2013, CCS.

[58]  Elissa M. Redmiles,et al.  Where is the Digital Divide?: A Survey of Security, Privacy, and Socioeconomics , 2017, CHI.

[59]  Eyal Péer,et al.  Better Late ( r ) than Never : Increasing CyberSecurity Compliance by Reducing Present Bias , 2018 .

[60]  Cedric Croft,et al.  Assessing the Difficulty of Reading Materials: The Noun Frequency Method. Revised Edition. , 1975 .

[61]  Rick Wash,et al.  Identifying patterns in informal sources of security information , 2015, J. Cybersecur..

[62]  James Nicholson,et al.  "If It's Important It Will Be A Headline": Cybersecurity Information Seeking in Older Adults , 2019, CHI.

[63]  Earl F. Rankin,et al.  Comparable Cloze and Multiple-Choice Comprehension Test Scores. , 1969 .

[64]  Abhijit Dasgupta,et al.  Practical Data Science Cookbook , 2014 .

[65]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[66]  Wilson L. Taylor,et al.  Recent Developments in the Use of “Cloze Procedure” , 1956 .

[67]  Tudor Dumitras,et al.  Asking for a Friend: Evaluating Response Biases in Security User Studies , 2018, CCS.

[68]  Yiming Yang,et al.  Deep Learning for Extreme Multi-label Text Classification , 2017, SIGIR.