Inform Product Change through Experimentation with Data-Driven Behavioral Segmentation

Online controlled experimentation is widely adopted for evaluating new features in the rapid development cycle for web products and mobile applications. Measurement on overall experiment sample is a common practice to quantify the overall treatment effect. In order to understand why the treatment effect occurs in a certain way, segmentation becomes a valuable approach to a finer analysis of experiment results. This paper introduces a framework for creating and utilizing user behavioral segments in online experimentation. By using the data of user engagement with individual product components as input, this method defines segments that are closely related to the features being evaluated in the product development cycle. With a real-world example we demonstrate that the analysis with such behavioral segments offered deep, actionable insights that successfully informed product decision making.

[1]  Donald W. Bouldin,et al.  A Cluster Separation Measure , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  M. Wedel,et al.  Market Segmentation: Conceptual and Methodological Foundations , 1997 .

[3]  Anmol Bhasin,et al.  From Infrastructure to Culture: A/B Testing Challenges in Large Scale Social Networks , 2015, KDD.

[4]  Greg Hamerly,et al.  Learning the k in k-means , 2003, NIPS.

[5]  Newton M. Campos.  The Lean Startup: How today's entrepreneurs use continuous innovation to create radically successful businesses , 2014 .

[6]  Wen Zhang,et al.  How much can behavioral targeting help online advertising? , 2009, WWW '09.

[7]  Bamshad Mobasher,et al.  Web User Segmentation Based on a Mixture of Factor Analyzers , 2006, EC-Web.

[8]  Ron Kohavi,et al.  Unexpected results in online controlled experiments , 2011, SKDD.

[9]  Zhenyu Zhao,et al.  Online Experimentation Diagnosis and Troubleshooting Beyond AA Validation , 2016, 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[10]  Ron Kohavi,et al.  Seven pitfalls to avoid when running controlled experiments on the web , 2009, KDD.

[11]  Rainer Alt,et al.  IEEE/WIC/ACM International Conference on Web Intelligence , 2015, WI-IAT.

[12]  Marcelo Maia,et al.  Identifying user behavior in online social networks , 2008, SocialNets '08.

[13]  Huizhi Xie,et al.  Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix , 2016, KDD.

[14]  Robert K. Young,et al.  Introduction to design and analysis: A student's handbook. , 1980 .

[15]  Ron Kohavi,et al.  Seven rules of thumb for web site experimenters , 2014, KDD.

[16]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[17]  L. Wasserman,et al.  A Reference Bayesian Test for Nested Hypotheses and its Relationship to the Schwarz Criterion , 1995 .

[18]  Ron Kohavi,et al.  Practical guide to controlled experiments on the web: listen to your customers not to the hippo , 2007, KDD '07.

[19]  Doug Beeferman,et al.  Agglomerative clustering of a search engine query log , 2000, KDD '00.

[20]  Peter Hofgesang.  Web Personalisation Through Incremental Individual Profiling and Support-based User Segmentation , 2007, IEEE/WIC/ACM International Conference on Web Intelligence (WI'07).

[21]  Ian T. Jolliffe,et al.  Principal Component Analysis , 2002, International Encyclopedia of Statistical Science.

[22]  Chaojun Lu,et al.  Topic-Based User Segmentation for Online Advertising with Latent Dirichlet Allocation , 2010, ADMA.

[23]  Douglas A. Reynolds.  Gaussian Mixture Models , 2009, Encyclopedia of Biometrics.

[24]  Ron Kohavi,et al.  Online controlled experiments at large scale , 2013, KDD.

[25]  Yan He,et al.  A Probabilistic, Mechanism-Indepedent Outlier Detection Method for Online Experimentation , 2017, 2017 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[26]  Ron Kohavi,et al.  Controlled experiments on the web: survey and practical guide , 2009, Data Mining and Knowledge Discovery.

[27]  Eamonn J. Keogh,et al.  Curse of Dimensionality , 2010, Encyclopedia of Machine Learning.