Measuring Player Retention and Monetization Using the Mean Cumulative Function

Game analytics supports game development by providing direct quantitative feedback about player experience. Player retention and monetization have become central business statistics in free-to-play game development. Total playtime and lifetime value in particular are central benchmarks, but many metrics have been used for this purpose. However, game developers often want to perform analytics in a timely manner before all users have churned from the game. This causes data censoring, which makes many metrics biased. In this article, we introduce how the mean cumulative function (MCF) can be used to measure metrics from censored data. Statistical tools based on the MCF allow game developers to determine whether a given change improves a game or whether a game is good enough for public release. The MCF is a general tool that estimates the expected value of a metric for any data set and does not rely on a model for the data. We demonstrate the advantages of this approach on a real in-development free-to-play mobile game Hipster Sheep.

[1]  W. Marsden.  I and J , 2012 .

[2]  Boi Faltings,et al.  Hidden Markov models for churn prediction , 2015, 2015 SAI Intelligent Systems Conference (IntelliSys).

[3]  R J Cook,et al.  Robust tests for treatment comparisons based on recurrent event responses. , 1996, Biometrics.

[4]  B. Ripley,et al.  Robust Statistics , 2018, Wiley Series in Probability and Statistics.

[5]  Ørnulf Borgan,et al.  Confidence intervals and confidence bands for the cumulative hazard rate function and their small sample properties , 1986 .

[6]  Richard J. Cook,et al.  The Statistical Analysis of Recurrent Events , 2007 .

[7]  Jukka Heikkonen,et al.  Modelling user retention in mobile games , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[8]  Debanjan Saha,et al.  A long-term study of a popular MMORPG , 2007, NetGames '07.

[9]  William Q. Meeker,et al.  Recurrent Events Data Analysis for Product Repairs, Disease Recurrences, and Other Applications , 2003, Technometrics.

[10]  Peng Gao,et al.  Churn prediction for high-value players in casual social games , 2014, 2014 IEEE Conference on Computational Intelligence and Games.

[11]  Jianwen Cai,et al.  Some Graphical Displays and Marginal Regression Analyses for Recurrent Failure Times and Time Dependent Covariates , 1993 .

[12]  R. Prentice,et al.  Commentary on Andersen and Gill's "Cox's Regression Model for Counting Processes: A Large Sample Study" , 1982 .

[13]  Jerald F. Lawless,et al.  Statistical Methods in Reliability , 1983 .

[14]  Jerald F. Lawless,et al.  The Analysis of Recurrent Events for Multiple Subjects , 1995 .

[15]  Christian Bauckhage,et al.  The Playtime Principle: Large-scale cross-games interest modeling , 2014, 2014 IEEE Conference on Computational Intelligence and Games.

[16]  Wayne Nelson,et al.  Graphical Analysis of System Repair Data , 1988 .

[17]  N Doganaksoy,et al.  A Method to Compare Two Samples of Recurrence Data , 1998, Lifetime data analysis.

[18]  Rüdiger Zarnekow,et al.  Counting Customers in Mobile Business - The Case of Free to Play , 2015, PACIS.

[19]  Jukka Heikkonen,et al.  A/B-Test of Retention and Monetization Using the Cox Model , 2017, AIIDE.

[20]  Kyung-Joong Kim,et al.  Game Data Mining Competition on Churn Prediction and Survival Analysis Using Commercial Game Log Data , 2018, IEEE Transactions on Games.

[21]  Andrew Nealen,et al.  Exploring Game Space Using Survival Analysis , 2015, FDG.

[22]  Fabio Zambetta,et al.  Predicting player churn in destiny: A Hidden Markov models approach to predicting player departure in a major online game , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[23]  Timothy Victor Fields,et al.  Game Industry Metrics Terminology and Analytics Case Study , 2013, Game Analytics, Maximizing the Value of Player Data.

[24]  Aaas News,et al.  Book Reviews , 1893, Buffalo Medical and Surgical Journal.

[25]  Cristina V. Lopes,et al.  Gate Me If You Can: The Impact of Gating Mechanics on Retention and Revenues in Jelly Splash , 2015, FDG.

[26]  Kuan-Ta Chen,et al.  Effect of Network Quality on Player Departure Behavior in Online Games , 2009, IEEE Transactions on Parallel and Distributed Systems.

[27]  Zhiliang Ying,et al.  Semiparametric regression for the mean and rate functions of recurrent events , 2000 .

[28]  Niels Keiding,et al.  Linear Nonparametric Tests for Comparison of Counting Processes, with Applications to Censored Survival Data, Correspondent Paper , 1982 .

[29]  Niels Keiding,et al.  Linear nonparametric tests for comparison of counting processes, with applications to censored survival data : (preprint) , 1981 .

[30]  M. R. Leadbetter.  Poisson Processes , 2011, International Encyclopedia of Statistical Science.

[31]  O. Aalen.  Nonparametric Inference for a Family of Counting Processes , 1978 .

[32]  Bernard Altshuler,et al.  Theory for the measurement of competing risks in animal experiments , 1970 .

[33]  Wayne Nelson,et al.  Confidence Limits for Recurrence Data—Applied to Cost or Number of Product Repairs , 1995 .

[34]  Christian Bauckhage,et al.  Predicting player churn in the wild , 2014, 2014 IEEE Conference on Computational Intelligence and Games.

[35]  Daniel Zappala,et al.  Condensing Steam: Distilling the Diversity of Gamer Behavior , 2016, Internet Measurement Conference.

[36]  Tapio Pahikkala,et al.  Playtime Measurement With Survival Analysis , 2017, IEEE Transactions on Games.

[37]  Bruce G. S. Hardie,et al.  Probability Models for Customer-Base Analysis , 2009 .

[38]  Agathe Guilloux,et al.  Design influence on player retention: A method based on time varying survival analysis , 2016, 2016 IEEE Conference on Computational Intelligence and Games (CIG).

[39]  Christian Bauckhage,et al.  Predicting Purchase Decisions in Mobile Free-to-Play Games , 2015, AIIDE.

[40]  Wayne Nelson,et al.  Hazard Plotting for Incomplete Failure Data , 1969 .

[41]  Ørnulf Borgan,et al.  Counting process models for life history data: a review , 1984 .

[42]  Anna Guitart,et al.  Churn Prediction in Mobile Social Games: Towards a Complete Assessment Using Survival Ensembles , 2016, 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA).

[43]  Cristina V. Lopes,et al.  Retention and progression: Seven months in World of Warcraft , 2014, FDG.

[44]  Wayne Nelson.  Theory and applications of hazard plotting for censored failure data , 2000 .

[45]  Wayne Nelson.  Theory and Applications of Hazard Plotting for Censored Failure Data , 2000, Technometrics.

[46]  Arnav Jhala,et al.  Using Data Mining to Model Player Experience , 2011 .

[47]  Kuan-Ta Chen,et al.  An analysis of WoW players' game hours , 2008, NETGAMES.