The Benefits of Controlled Experimentation at Scale

Online controlled experiments (for example A/B tests) are increasingly being performed to guide product development and accelerate innovation in online software product companies. The benefits of controlled experiments have been shown in many cases with incremental product improvement as the objective. In this paper, we demonstrate that the value of controlled experimentation at scale extends beyond this recognized scenario. Based on an exhaustive and collaborative case study in a large software-intensive company with highly developed experimentation culture, we inductively derive the benefits of controlled experimentation. The contribution of our paper is twofold. First, we present a comprehensive list of benefits and illustrate our findings with five case examples of controlled experiments conducted at Microsoft. Second, we provide guidance on how to achieve each of the benefits. With our work, we aim to provide practitioners in the online domain with knowledge on how to use controlled experimentation to maximize the benefits on the portfolio, product and team level.

[1]  Per Runeson,et al.  Guidelines for conducting and reporting case study research in software engineering , 2009, Empirical Software Engineering.

[2]  Robert C. Martin Agile Software Development, Principles, Patterns, and Practices , 2002 .

[3]  Steven M. Drucker,et al.  The Bones of the System: A Case Study of Logging and Telemetry at Microsoft , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C).

[4]  Ron Kohavi,et al.  Online controlled experiments at large scale , 2013, KDD.

[5]  V. Ridgway Dysfunctional Consequences of Performance Measurements , 1956 .

[6]  Jürgen Münch,et al.  Raising the odds of success: the current state of experimentation in product development , 2016, Inf. Softw. Technol..

[7]  Alex Deng,et al.  Data-Driven Metric Development for Online Controlled Experiments: Seven Lessons Learned , 2016, KDD.

[8]  Aleksander Fabijan Developing the right features: the role and impact of customer and product data in software product development , 2016 .

[9]  Henk Thierry,et al.  How leaders stimulate employee learning: A leader-member exchange approach , 2010 .

[10]  Jan Bosch,et al.  Towards Continuous Customer Validation: A Conceptual Model for Combining Qualitative Customer Feedback with Quantitative Customer Observation , 2015, ICSOB.

[11]  Dong Woo Kim,et al.  A Dirty Dozen: Twelve Common Metric Interpretation Pitfalls in Online Controlled Experiments , 2017, KDD.

[12]  Ron Kohavi,et al.  Pitfalls of long-term online controlled experiments , 2016, 2016 IEEE International Conference on Big Data (Big Data).

[13]  James S. Noble,et al.  The changing basis of performance measurement , 1996 .

[14]  Colin Robson,et al.  Real World Research: A Resource for Social Scientists and Practitioner-Researchers , 1993 .

[15]  Ron Kohavi,et al.  Seven pitfalls to avoid when running controlled experiments on the web , 2009, KDD.

[16]  Jan Bosch,et al.  Time to Say 'Good Bye': Feature Lifecycle , 2016, 2016 42th Euromicro Conference on Software Engineering and Advanced Applications (SEAA).

[17]  E. A. Locke,et al.  Building a practically useful theory of goal setting and task motivation. A 35-year odyssey. , 2002, The American psychologist.

[18]  Fabian Fagerholm,et al.  Building blocks for continuous experimentation , 2014, RCoSE 2014.

[19]  Lech Madeyski,et al.  Software Engineering Needs Agile Experimentation: A New Practice and Supporting Tool , 2016, KKIO Software Engineering Conference.

[20]  P. Schroeder The Goal: A Process of Ongoing Improvement , 1994 .

[21]  Jan Bosch,et al.  The HYPEX Model: From Opinions to Data-Driven Software Development , 2014, Continuous Software Engineering.

[22]  Jan Bosch,et al.  User involvement throughout the innovation process in high-tech industries , 2015 .

[23]  Jürgen Münch,et al.  The RIGHT Model for Continuous Experimentation , 2017, Software Engineering.

[24]  Ashish Agarwal,et al.  Overlapping experiment infrastructure: more, better, faster experimentation , 2010, KDD.

[25]  Ron Kohavi,et al.  Trustworthy online controlled experiments: five puzzling outcomes explained , 2012, KDD.

[26]  Michael S. Bernstein,et al.  Designing and deploying online field experiments , 2014, WWW.

[27]  Ron Kohavi,et al.  Responsible editor: R. Bayardo. , 2022 .

[28]  Jürgen Münch,et al.  Continuous Experimentation in the B2B Domain: A Case Study , 2014, 2015 IEEE/ACM 2nd International Workshop on Rapid Continuous Software Engineering.

[29]  Kai Petersen,et al.  Waste and Lead Time Reduction in a Software Product Customization Process with Value Stream Maps , 2010, 2010 21st Australian Software Engineering Conference.

[30]  Ron Kohavi,et al.  Online Controlled Experiments and A / B Tests , 2015 .

[31]  Miryung Kim,et al.  The Emerging Role of Data Scientists on Software Development Teams , 2016, 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE).

[32]  E. A. Locke,et al.  Goal setting and task performance: 1969–1980. , 1981 .

[33]  George Castellion Do It Wrong Quickly: How the Web Changes the Old Marketing Rules by Mike Moran , 2008 .

[34]  Dan Siroker,et al.  A/B Testing: The Most Powerful Way to Turn Clicks Into Customers , 2013 .

[35]  Jan Bosch,et al.  From Opinions to Data-Driven Software R&D: A Multi-case Study on How to Close the 'Open Loop' Problem , 2014, 2014 40th EUROMICRO Conference on Software Engineering and Advanced Applications.

[36]  Hilary Hutchinson,et al.  Measuring the user experience on a large scale: user-centered metrics for web applications , 2010, CHI.

[37]  Xian Wu,et al.  Measuring Metrics , 2016, CIKM.

[38]  Jan Bosch,et al.  The Evolution of Continuous Experimentation in Software Product Development: From Data to a Data-Driven Organization at Scale , 2017, 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE).

[39]  Fabien Girardin,et al.  When User Experience Designers Partner with Data Scientists , 2017, AAAI Spring Symposia.

[40]  Diane Tang,et al.  Focusing on the Long-term: It's Good for Users and Business , 2015, KDD.

[41]  Richard Craig Van Nostrand,et al.  Design of Experiments Using the Taguchi Approach: 16 Steps to Product and Process Improvement , 2002, Technometrics.

[42]  Jan Bosch,et al.  Climbing the "Stairway to Heaven" -- A Mulitiple-Case Study Exploring Barriers in the Transition from Agile Development towards Continuous Deployment of Software , 2012, 2012 38th Euromicro Conference on Software Engineering and Advanced Applications.

[43]  Jan Bosch,et al.  Customer Feedback and Data Collection Techniques in Software R&D: A Literature Review , 2015, ICSOB.

[44]  Ron Kohavi,et al.  Seven rules of thumb for web site experimenters , 2014, KDD.

[45]  Brijesh Singh,et al.  The Lean Startup:How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses , 2016 .

[46]  Pasi Kuvaja,et al.  Continuous deployment of software intensive products and services: A systematic mapping study , 2017, J. Syst. Softw..