Towards Automated A/B Testing

User-intensive software, such asWeb andmobile applications, heavily depends on the interactions with large and unknown populations of users. Knowing the preferences and behaviors of these populations is crucial for the success of this class of systems. A/B testing is an increasingly popular technique that supports the iterative development of userintensive software based on controlled experiments performed on live users. However, as currently performed, A/B testing is a time consuming, error prone and costly manual activity. In this paper, we investigate a novel approach to automate A/B testing. More specifically, we rephrase A/B testing as a search-based software engineering problem and we propose an initial approach that supports automated A/B testing through aspect-oriented programming and genetic algorithms.

[1]  Cristina V. Lopes,et al.  Aspect-oriented programming , 1999, ECOOP Workshops.

[2]  Ron Kohavi,et al.  Trustworthy online controlled experiments: five puzzling outcomes explained , 2012, KDD.

[3]  Gregor Kiczales,et al.  Aspect-oriented programming , 2001, ESEC/FSE-9.

[4]  Outi Räihä,et al.  A survey on search-based software design , 2010, Comput. Sci. Rev..

[5]  Ron Kohavi,et al.  Practical guide to controlled experiments on the web: listen to your customers not to the hippo , 2007, KDD '07.

[6]  Ron Kohavi,et al.  Seven pitfalls to avoid when running controlled experiments on the web , 2009, KDD.

[7]  Wasif Afzal,et al.  A systematic review of search-based testing for non-functional system properties , 2009, Inf. Softw. Technol..

[8]  Ron Kohavi,et al.  Unexpected results in online controlled experiments , 2011, SKDD.

[9]  Mark Harman,et al.  Search-based software engineering , 2001, Inf. Softw. Technol..

[10]  K. Deb An Efficient Constraint Handling Method for Genetic Algorithms , 2000 .

[11]  Satoshi Matsuoka,et al.  ECOOP'97 — Object-Oriented Programming , 1997, Lecture Notes in Computer Science.

[12]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[13]  Yuanyuan Zhang,et al.  Search Based Requirements Optimisation: Existing Work and Challenges , 2008, REFSQ.

[14]  Ron Kohavi,et al.  Online controlled experiments at large scale , 2013, KDD.

[15]  Mark Kent O'Keeffe,et al.  Search-based software maintenance , 2006, Conference on Software Maintenance and Reengineering (CSMR'06).