Best Practices for Notification Studiesfor Security and Privacy Issues on the Internet

Researchers help operators of vulnerable and non-compliant internet services by individually notifying them about security and privacy issues uncovered in their research. To improve efficiency and effectiveness of such efforts, dedicated notification studies are imperative. As of today, there is no comprehensive documentation of pitfalls and best practices for conducting such notification studies, which limits validity of results and impedes reproducibility. Drawing on our experience with such studies and guidance from related work, we present a set of guidelines and practical recommendations, including initial data collection, sending of notifications, interacting with the recipients, and publishing the results. We note that future studies can especially benefit from extensive planning and automation of crucial processes, i. e., activities that take place well before the first notifications are sent.

[1]  Tyler Moore,et al.  Understanding the Role of Sender Reputation in Abuse Reporting and Cleanup , 2015, WEIS.

[2]  Aurélien Francillon,et al.  The role of web hosting providers in detecting compromised websites , 2013, WWW '13.

[3]  Samaneh Tajalizadehkhoob,et al.  Tell Me You Fixed It: Evaluating Vulnerability Notifications via Quarantine Networks , 2019, 2019 IEEE European Symposium on Security and Privacy (EuroS&P).

[4]  Wouter Joosen,et al.  Tranco: A Research-Oriented Top Sites Ranking Hardened Against Manipulation , 2018, NDSS.

[5]  Janice Y. Tsai,et al.  You Needn't Build That: Reusable Ethics-Compliance Infrastructure for Human Subjects Research , 2013 .

[6]  Tyler Moore,et al.  Do Malware Reports Expedite Cleanup? An Experimental Study , 2012, CSET.

[7]  Michael Backes,et al.  Hey, You Have a Problem: On the Feasibility of Large-Scale Web Vulnerability Notification , 2016, USENIX Security Symposium.

[8]  Narseo Vallina-Rodriguez,et al.  A Long Way to the Top: Significance, Structure, and Stability of Internet Top Lists , 2018, Internet Measurement Conference.

[9]  Michael Backes,et al.  Didn't You Hear Me? - Towards More Successful Web Vulnerability Notifications , 2018, NDSS.

[10]  Katie Shilton,et al.  Beyond the Belmont Principles: Ethical Challenges, Practices, and Beliefs in the Online Data Research Community , 2016, CSCW.

[11]  Vern Paxson,et al.  The Matter of Heartbleed , 2014, Internet Measurement Conference.

[12]  Adrienne Porter Felt,et al.  Fixing HTTPS Misconfigurations at Scale: An Experiment with Security Notifications , 2019 .

[13]  Matthias Hollick,et al.  On the Difficulties of Incentivizing Online Privacy through Transparency: A Qualitative Survey of the German Health Insurance Market , 2019, Wirtschaftsinformatik.

[14]  Jacob Leon Kröger,et al.  How do app vendors respond to subject access requests? A longitudinal privacy study on iOS and Android Apps , 2020, ARES.

[15]  Janice Y. Tsai,et al.  Tell Me Lies: A Methodology for Scientif- ically Rigorous Security User Studies , 2010 .

[16]  Vern Paxson,et al.  Remedying Web Hijacking: Notification Effectiveness and Webmaster Comprehension , 2016, WWW.

[17]  Matthias Hollick,et al.  Snail Mail Beats Email Any Day:On Effective Operator Security Notifications in the Internet , 2021, ARES.

[18]  Arvind Narayanan,et al.  Online Tracking: A 1-million-site Measurement and Analysis , 2016, CCS.

[19]  W. Douglas Maughan,et al.  The Menlo Report , 2012, IEEE Security & Privacy.

[20]  Matthias Hollick,et al.  Effective Notification Campaigns on the Web: A Matter of Trust, Framing, and Support , 2020, USENIX Security Symposium.

[21]  Stefan Savage,et al.  You've Got Vulnerability: Exploring Effective Vulnerability Notifications , 2016, USENIX Security Symposium.

[22]  Nicolas P. Rougier,et al.  Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions , 2017, Front. Neuroinform..

[23]  Eric Wustrow,et al.  ZMap: Fast Internet-wide Scanning and Its Security Applications , 2013, USENIX Security Symposium.

[24]  Christian Rossow,et al.  Exit from Hell? Reducing the Impact of Amplification DDoS Attacks , 2014, USENIX Security Symposium.

[25]  Max Maass,et al.  PrivacyScore: Analyse von Webseiten auf Sicherheits- und Privatheitsprobleme , 2017, GI-Jahrestagung.

[26]  Kangjie Lu,et al.  IEEE S&P’21 Program Committee Statement Regarding The “Hypocrite Commits” Paper , 2021 .

[27]  Samaneh Tajalizadehkhoob,et al.  Let Me Out! Evaluating the Effectiveness of Quarantining Compromised Users in Walled Gardens , 2018, SOUPS @ USENIX Security Symposium.

[28]  Kensuke Fukuda,et al.  Profiling internet scanners: Spatiotemporal structures and measurement ethics , 2017, 2017 Network Traffic Measurement and Analysis Conference (TMA).

[29]  Kat Krol,et al.  Towards Robust Experimental Design for User Studies in Security and Privacy , 2016 .

[30]  M. V. Eeten,et al.  Make notifications great again: learning how to notify in the age of large-scale vulnerability scanning , 2017 .

[31]  Klaus Wehrle,et al.  The Dagstuhl beginners guide to reproducibility for experimental networking research , 2019, CCRV.

[32]  Katharina Krombholz,et al.  Investigating System Operators' Perspective on Security Misconfigurations , 2018, CCS.

[33]  Salvatore J. Stolfo,et al.  Reflections on the engineering and operation of a large-scale embedded device vulnerability scanner , 2011, BADGERS '11.