The Workshop on Big Data Benchmarking (WBDB2012), held on May 8-9, 2012 in San Jose, CA, served as an incubator for several promising approaches to define a big data benchmark standard for industry. Through an open forum for discussions on a number of issues related to big data benchmarking—including definitions of big data terms, benchmark processes and auditing — the attendees were able to extend their own view of big data benchmarking as well as communicate their own ideas, which ultimately led to the formation of small working groups to continue collaborative work in this area. In this paper, we summarize the discussions and outcomes from this first workshop, which was attended by about 60 invitees representing 45 different organizations, including industry and academia. Workshop attendees were selected based on their experience and expertise in the areas of management of big data, database systems, performance benchmarking, and big data applications. There was consensus among participants about both the need and the opportunity for defining benchmarks to capture the end-to-end aspects of big data applications. Following the model of TPC benchmarks, it was felt that big data benchmarks should not only include metrics for performance, but also price/performance, along with a sound foundation for fair comparison through audit mechanisms. Additionally, the benchmarks should consider several costs relevant to big data systems including total cost of acquisition, setup cost, and the total cost of ownership, including energy cost. The second Workshop on Big Data Benchmarking will be held in December 2012 in Pune, India, and the third meeting is being planned for July 2013 in Xi’an, China.
[1]
Meikel Pöss,et al.
TPC-DS, taking decision support benchmarking to the next level
,
2002,
SIGMOD '02.
[2]
Raghunath Othayoth Nambiar,et al.
Why You Should Run TPC-DS: A Workload Analysis
,
2007,
VLDB.
[3]
Tilmann Rabl,et al.
A Data Generator for Cloud-Scale Benchmarking
,
2010,
TPCTC.
[4]
Meikel Pöss,et al.
New TPC benchmarks for decision support and web commerce
,
2000,
SGMD.
[5]
Karl Huppler.
Price and the TPC
,
2010,
TPCTC.
[6]
Lin Xiao,et al.
YCSB++: benchmarking and performance debugging advanced features in scalable table stores
,
2011,
SoCC.
[7]
Trish Hogan.
Overview of TPC Benchmark E: The Next Generation of OLTP Benchmarks
,
2009,
TPCTC.
[8]
Raghunath Othayoth Nambiar,et al.
The making of TPC-DS
,
2006,
VLDB.
[9]
O’Tar T. Norwood.
TORONTO
,
1993,
International Society of Hair Restoration Surgery.