Historically, the process of synchronizing a decision support system with data from operational systems has been referred to as Extract, Transform, Load (ETL) and the tools supporting such process have been referred to as ETL tools. Recently, ETL was replaced by the more comprehensive acronym, data integration (DI). DI describes the process of extracting and combining data from a variety of data source formats, transforming that data into a unified data model representation and loading it into a data store. This is done in the context of a variety of scenarios, such as data acquisition for business intelligence, analytics and data warehousing, but also synchronization of data between operational applications, data migrations and conversions, master data management, enterprise data sharing and delivery of data services in a service-oriented architecture context, amongst others. With these scenarios relying on up-to-date information it is critical to implement a highly performing, scalable and easy to maintain data integration system. This is especially important as the complexity, variety and volume of data is constantly increasing and performance of data integration systems is becoming very critical. Despite the significance of having a highly performing DI system, there has been no industry standard for measuring and comparing their performance. The TPC, acknowledging this void, has released TPC-DI, an innovative benchmark for data integration. This paper motivates the reasons behind its development, describes its main characteristics including workload, run rules, metric, and explains key decisions.
[1]
Tilmann Rabl,et al.
Parallel data generation for performance analysis of large, complex RDBMS
,
2011,
DBTest '11.
[2]
Umeshwar Dayal,et al.
Benchmarking ETL Workflows
,
2009,
TPCTC.
[3]
Ralph Kimball,et al.
The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling
,
1996
.
[4]
Raghunath Othayoth Nambiar,et al.
Why You Should Run TPC-DS: A Workload Analysis
,
2007,
VLDB.
[5]
Panos Vassiliadis,et al.
Towards a Benchmark for ETL Workflows
,
2007,
QDB.
[6]
Tilmann Rabl,et al.
A Data Generator for Cloud-Scale Benchmarking
,
2010,
TPCTC.
[7]
Tilmann Rabl,et al.
Efficient update data generation for DBMS benchmarks
,
2012,
ICPE '12.