Methodologies for a Comprehensive Approach to Measuring the Performance of Decision Support Systems

This work develops methodologies for a comprehensive approach to measuring the performance of decision support systems in the domains of Internet of Things (IoT), data integration and SQL-based big data systems. These methodologies were standardized into the specifications of three industry standard performance benchmarks, published by the Transaction Processing Performance Council: TPCx-IoT for IoT gateway systems, TPC-DI for data integration systems, and TPC-DS for SQL-based big data systems.

[1]  Kiyoung Kim,et al.  MRBench: A Benchmark for MapReduce Framework , 2008, 2008 14th IEEE International Conference on Parallel and Distributed Systems.

[2]  Tilmann Rabl,et al.  Parallel data generation for performance analysis of large, complex RDBMS , 2011, DBTest '11.

[3]  Raghunath Othayoth Nambiar,et al.  Introducing TPCx-HS: The First Industry Standard for Benchmarking Big Data Systems , 2014, TPCTC.

[4]  Manish Marwah,et al.  IoTAbench: an Internet of Things Analytics Benchmark , 2015, ICPE.

[5]  Tilmann Rabl,et al.  Analysis of TPCx-IoT: The First Industry Standard Benchmark for IoT Gateway Systems , 2018, 2018 IEEE 34th International Conference on Data Engineering (ICDE).

[6]  Michael Stonebraker,et al.  A comparison of approaches to large-scale data analysis , 2009, SIGMOD Conference.

[7]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[8]  Jie Huang,et al.  The HiBench benchmark suite: Characterization of the MapReduce-based data analysis , 2010, 2010 IEEE 26th International Conference on Data Engineering Workshops (ICDEW 2010).

[9]  Russell Binions,et al.  Metal Oxide Semi-Conductor Gas Sensors in Environmental Monitoring , 2010, Sensors.

[10]  Adam Silberstein,et al.  Benchmarking cloud serving systems with YCSB , 2010, SoCC '10.

[11]  Raghunath Othayoth Nambiar,et al.  Why You Should Run TPC-DS: A Workload Analysis , 2007, VLDB.

[12]  Umeshwar Dayal,et al.  Benchmarking ETL Workflows , 2009, TPCTC.

[13]  Alain Crolotte.  Issues in Benchmark Metric Selection , 2009, TPCTC.

[14]  Li Zhang,et al.  SparkBench: a comprehensive benchmarking suite for in memory data analytic platform Spark , 2015, Conf. Computing Frontiers.

[15]  Gordon B. Davis,et al.  Management information systems : conceptual foundations, structure, and development , 1985 .

[16]  Tilmann Rabl,et al.  BigBench Specification V0.1 - BigBench: An Industry Standard Benchmark for Big Data Analytics , 2012, WBDB.

[17]  Meikel Pöss,et al.  Generating Thousand Benchmark Queries in Seconds , 2004, VLDB.

[18]  Timothy G. Armstrong,et al.  LinkBench: a database benchmark based on the Facebook social graph , 2013, SIGMOD '13.

[19]  Tilmann Rabl,et al.  Big Data Benchmark Compendium , 2015, TPCTC.

[20]  Tilmann Rabl,et al.  A Data Generator for Cloud-Scale Benchmarking , 2010, TPCTC.

[21]  Brian W. Barrett,et al.  Introducing the Graph 500 , 2010 .

[22]  Meikel Pöss,et al.  TPC-DS, taking decision support benchmarking to the next level , 2002, SIGMOD '02.

[23]  Tilmann Rabl,et al.  TPC-DI: The First Industry Benchmark for Data Integration , 2014, Proc. VLDB Endow..

[24]  Raghunath Othayoth Nambiar,et al.  The making of TPC-DS , 2006, VLDB.

[25]  Ralph Kimball,et al.  The Data Warehouse Toolkit: The Complete Guide to Dimensional Modeling , 1996 .

[26]  Karl Huppler,et al.  TPC Express - A New Path for TPC Benchmarks , 2013, TPCTC.

[27]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[28]  Ravi Kumar,et al.  Pig latin: a not-so-foreign language for data processing , 2008, SIGMOD Conference.

[29]  Robert Tibshirani,et al.  Estimating the number of clusters in a data set via the gap statistic , 2000 .

[30]  Tilmann Rabl,et al.  Efficient update data generation for DBMS benchmarks , 2012, ICPE '12.

[31]  Karl Huppler,et al.  The Art of Building a Good Benchmark , 2009, TPCTC.

[32]  Scott Shenker,et al.  Shark: SQL and rich analytics at scale , 2012, SIGMOD '13.

[33]  Panos Vassiliadis,et al.  Towards a Benchmark for ETL Workflows , 2007, QDB.

[34]  Tilmann Rabl,et al.  Analysis of TPC-DS: the first standard benchmark for SQL-based big data systems , 2017, SoCC.

[35]  Omri Serlin.  The History of DebitCredit and the TPC , 1991, The Benchmark Handbook.

[36]  Pete Wyckoff,et al.  Hive - A Warehousing Solution Over a Map-Reduce Framework , 2009, Proc. VLDB Endow..

[37]  Wilson C. Hsieh,et al.  Bigtable: A Distributed Storage System for Structured Data , 2006, TOCS.

[38]  Meikel Pöss,et al.  New TPC benchmarks for decision support and web commerce , 2000, SGMD.

[39]  Karl Huppler.  Price and the TPC , 2010, TPCTC.

[40]  Carrie Ballinger.  TPC-D: Benchmarking for Decision Support , 1993, The Benchmark Handbook.

[41]  Jim Gray,et al.  The Benchmark Handbook for Database and Transaction Systems , 1993 .

[42]  Meikel Pöss,et al.  MUDD: a multi-dimensional data generator , 2004, WOSP '04.

[43]  Michael Stonebraker,et al.  A New Direction for TPC? , 2009, TPCTC.

[44]  Carlo Curino,et al.  Apache Hadoop YARN: yet another resource negotiator , 2013, SoCC.