Lessons Learned from Using an Online Platform to Conduct Large-Scale, Online Controlled Security Experiments with Software Developers

Security and privacy researchers are increasingly conducting controlled experiments focusing on IT professionals, such as software developers and system administrators. These professionals are typically more difficult to recruit than general end-users. In order to allow for distributed recruitment of IT professionals for security user studies, we designed Developer Observatory, a browser-based virtual laboratory platform that enables controlled programming experiments while retaining most of the observational power of lab studies. The Developer Observatory can be used to conduct largescale, reliable online programming studies with reasonable external validity. We report on our experiences and lessons learned from two controlled programming experiments (n>200) conducted using Developer Observatory.

[1]  Michelle L. Mazurek,et al.  You are Not Your Developer, Either: A Research Agenda for Usable Security and Privacy Research Beyond End Users , 2016, 2016 IEEE Cybersecurity Development (SecDev).

[2]  Andrew Ruef,et al.  Build It, Break It, Fix It: Contesting Secure Development , 2016, CCS.

[3]  José M. Fernandez,et al.  Computer Security Clinical Trials: Lessons Learned from a 4-month Pilot Study , 2014, CSET.

[4]  Michael Backes,et al.  You Get Where You're Looking for: The Impact of Information Sources on Code Security , 2016, 2016 IEEE Symposium on Security and Privacy (SP).

[5]  Nicolas Christin,et al.  Security Behavior Observatory: Infrastructure for Long-term Monitoring of Client Machines (CMU-CyLab-14-009) , 2014 .

[6]  Chunming Qiao,et al.  PhoneLab: A Large Programmable Smartphone Testbed , 2013, SENSEMINE@SenSys.

[7]  Rick Wash,et al.  Can People Self-Report Security Accurately?: Agreement Between Self-Report and Behavioral Measures , 2017, CHI.

[8]  Marcelo Masera,et al.  A Review of Available Software for the Creation of Testbeds for Internet Security Research , 2009, 2009 First International Conference on Advances in System Simulation.

[9]  Eileen Kraemer,et al.  Designing your Next Empirical Study on Program Comprehension , 2007, 15th IEEE International Conference on Program Comprehension (ICPC '07).

[10]  Terry V. Benzel.  The science of cyber security experimentation: the DETER project , 2011, ACSAC '11.

[11]  Simson L. Garfinkel,et al.  Comparing the Usability of Cryptographic APIs , 2017, 2017 IEEE Symposium on Security and Privacy (SP).

[12]  Sven Apel,et al.  Views on Internal and External Validity in Empirical Software Engineering , 2015, 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.

[13]  Michelle L. Mazurek,et al.  Security Developer Studies with GitHub Users: Exploring a Convenience Sample , 2017, SOUPS.