A Comparison of the Taguchi Method and Evolutionary optimization in Multivariate Testing

Multivariate testing has recently emerged as a promising technique in web interface design. In contrast to the standard A/B testing, multivariate approach aims at evaluating a large number of values in a few key variables systematically. The Taguchi method is a practical implementation of this idea, focusing on orthogonal combinations of values. It is the current state of the art in applications such as Adobe Target. This paper evaluates an alternative method: population-based search, i.e. evolutionary optimization. Its performance is compared to that of the Taguchi method in several simulated conditions, including an orthogonal one designed to favor the Taguchi method, and two realistic conditions with dependences between variables. Evolutionary optimization is found to perform significantly better especially in the realistic conditions, suggesting that it forms a good approach for web interface design and other related applications in the future.

[1]  Kalyanmoy Deb,et al.  Breaking the Billion-Variable Barrier in Real-World Optimization Using a Customized Evolutionary Algorithm , 2016, GECCO.

[2]  Peter S. Fader,et al.  Dynamic Conversion Behavior at E-Commerce Sites , 2004, Manag. Sci..

[3]  Danna Zhou,et al.  d. , 1934, Microbial pathogenesis.

[4]  Dan Boneh,et al.  On genetic algorithms , 1995, COLT '95.

[5]  Dario Floreano,et al.  Neuroevolution: from architectures to learning , 2008, Evol. Intell..

[6]  Risto Miikkulainen,et al.  Sentient Ascend: AI-Based Massively Multivariate Conversion Rate Optimization , 2018, AAAI.

[7]  Risto Miikkulainen,et al.  How to select a winner in evolutionary optimization? , 2017, SSCI.

[8]  Jürgen Branke,et al.  Evolutionary Optimization in Dynamic Environments , 2001, Genetic Algorithms and Evolutionary Computation.

[9]  Ron Kohavi,et al.  Online Controlled Experiments and A / B Tests , 2015 .

[10]  Risto Miikkulainen,et al.  Enhanced Optimization with Composite Objectives and Novelty Pulsation , 2018, GPTP.

[11]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[12]  Tim Ash,et al.  Landing Page Optimization: The Definitive Guide to Testing and Tuning for Conversions , 2008 .

[13]  Andries E. Brouwer,et al.  Orthogonal arrays of strength 3 and small run sizes , 2006 .

[14]  Jennifer Werfel,et al.  Orthogonal Arrays Theory And Applications , 2016 .

[15]  Ron Kohavi,et al.  The Surprising Power of Online Experiments , 2017 .

[16]  Phil J Hobbs,et al.  The Taguchi methodology as a statistical tool for biotechnological applications: A critical appraisal , 2008, Biotechnology journal.

[17]  Lih-Yuan Deng,et al.  Orthogonal Arrays: Theory and Applications , 1999, Technometrics.

[18]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[19]  田口 玄一,et al.  New Trends in Multivariate Diagnosis , 2001 .

[20]  Risto Miikkulainen,et al.  Conversion rate optimization through evolutionary computation , 2017, GECCO.