论文信息 - Problems and available solutions on the stage of Extract, Transform, and Loading in near real-time data warehousing (a literature study)

Problems and available solutions on the stage of Extract, Transform, and Loading in near real-time data warehousing (a literature study)

In the traditional ETL (Extract Transform Loading), refreshment of data warehouse must be done in off peak hours. It means that all operational and analysis stopped from their all activities. It cause the level of freshness of data in the data warehouse is not indicating the latest operational transaction. This problem is called by data latency. Near real time data warehousing is used to be a solution for this problem. It update data warehouse in near real time manner, immediately after change data detected in data source. Thus, data latency can be minimized. In development, near real time data warehousing have problems where previously not found on the traditional ETL. This paper aims to convey the problems and available solutions at each stage in the near real time data warehousing, i.e. extraction, transformation, and loading. The problems and available solutions are based on literature review from other research that focusing on near real time data warehousing problem.

Ardianto Wibowo | A. Wibowo

[1] A Min Tjoa,et al. Zero-Latency Data Warehousing for Heterogeneous Data Sources and Continuous Data Streams , 2003, iiWAS.

[2] Chen Lin,et al. Maintaining Internal Consistency of Report for Real-Time OLAP with Layer-Based View , 2011, APWeb.

[3] Jennifer Widom,et al. Models and issues in data stream systems , 2002, PODS.

[5] Shivani Saluja,et al. Refreshing Datawarehouse in Near Real-Time , 2012 .

[6] Jorge Bernardino,et al. Real-time data warehouse loading methodology , 2008, IDEAS '08.

[7] Gerald Weber,et al. An Event-Based Near Real-Time Data Integration Architecture , 2008, 2008 12th Enterprise Distributed Object Computing Conference Workshops.

[8] Kamal M. Kakish,et al. ETL Evolution for Real-Time Data Warehousing , 2012 .

[9] Carlos Roberto Valêncio,et al. Real Time Delta Extraction Based on Triggers to Support Data Warehousing , 2013, 2013 International Conference on Parallel and Distributed Computing, Applications and Technologies.

[10] Divyakant Agrawal. The Reality of Real-Time Business Intelligence , 2008, BIRTE.

[11] Robert J Davenport. Etl Vs Elt a Subjective View , 2009 .

[12] Janis Zuters. Near Real-Time Data Warehousing with Multi-stage Trickle and Flip , 2011, BIR.

[13] N. Karthikeyan,et al. From Data Warehouses to Streaming Warehouses: A Survey on the Challenges for Real-Time Data Warehousing and Available Solutions , 2013 .