Customer Relationship Management (CRM) is currently the fastest growing sector of enterprise software, estimated to increase to $36.5B worldwide by 2017. CRM technologies increasingly use data mining primitives across multiple applications. At the same time, the growth of big data has led to the evolution of an open source big data software stack (primarily powered by Apache software) that rivals traditional enterprise database (RDBMS) stacks. New technologies such as Kafka, Storm, HBase have significantly enriched this open source stack, alongside more established technologies such as Hadoop MapReduce and Mahout. Today, enterprises have a choice to make regarding which stack they will choose to power their big data applications. However, there are no published studies in literature on enterprise big data pipelines built using open source components supporting CRM. Specific questions that enterprises have include: how is the data processed and analyzed in such pipelines? What are the building blocks of such pipelines? How long does each step of this processing take? In this work, we answer these questions for a large scale (serving over a 100M customers) industrial CRM pipeline that incorporates data mining, and serves several applications. Our pipeline has, broadly, two parts. The first is a data gathering part that uses Kafka, Storm, and HBase. The second is a data mining part that uses Mahout and Hadoop MapReduce. We also provide timings for common tasks in the second part such as data preprocessing for machine learning, clustering, reservoir sampling, and frequent itemset extraction.
[1]
Qing He,et al.
Parallel Implementation of Apriori Algorithm Based on MapReduce
,
2012,
SNPD.
[2]
Beng Chin Ooi,et al.
The performance of MapReduce
,
2010,
Proc. VLDB Endow..
[3]
Keqiu Li,et al.
Optimized big data K-means clustering using MapReduce
,
2014,
The Journal of Supercomputing.
[4]
Ramakrishnan Srikant,et al.
Fast Algorithms for Mining Association Rules in Large Databases
,
1994,
VLDB.
[5]
ChenLu,et al.
A method for discovering clusters of e-commerce interest patterns using click-stream data
,
2015
.
[6]
Pete Wyckoff,et al.
Hive - A Warehousing Solution Over a Map-Reduce Framework
,
2009,
Proc. VLDB Endow..
[7]
Wilson C. Hsieh,et al.
Bigtable: A Distributed Storage System for Structured Data
,
2006,
TOCS.
[8]
Lu Chen,et al.
A method for discovering clusters of e-commerce interest patterns using click-stream data
,
2015,
Electron. Commer. Res. Appl..
[9]
Qing He,et al.
Parallel K-Means Clustering Based on MapReduce
,
2009,
CloudCom.
[10]
Roung-Shiunn Wu,et al.
Customer segmentation of multiple category data in e-commerce using a soft-clustering approach
,
2011,
Electron. Commer. Res. Appl..
[11]
Greg Linden,et al.
Amazon . com Recommendations Item-to-Item Collaborative Filtering
,
2001
.