Real Time Delta Extraction Based on Triggers to Support Data Warehousing

Nowadays large corporations require integrated data from diverse sources, leading to the use of data warehouse architectures for this purpose. To bypass problems related to the use of computational resources to process large volumes of data, an ETL (Extract, Transform and Load) technique with zero latency can be used, that works by constantly processing small data loads. Among the extraction techniques of the zero latency ETL are the use of logs, triggers, materialized views and timestamps. This paper proposes a structure capable of performing this task by means of triggers and a tool developed for the automatic generation of the SQL (Structured Query Language) code to create these trigger, besides showing its performance and comparing it to other techniques. Said method is relevant for the extraction of portions of selected information as it permits to combine conventional and real time ETL techniques.

[1]  Amit Ganatra,et al.  Speeding ETL Processing in Data Warehouses Using High-Performance Joins for Changed Data Capture (CDC) , 2010, 2010 International Conference on Advances in Recent Technologies in Communication and Computing.

[2]  Ruilian Hou Analysis and research on the difference between data warehouse and database , 2011, Proceedings of 2011 International Conference on Computer Science and Network Technology.

[3]  Guo-xiang Liu,et al.  The application of data warehouse in decision support system , 2012, 2012 2nd International Conference on Consumer Electronics, Communications and Networks (CECNet).

[4]  Panos Vassiliadis,et al.  Near Real Time ETL , 2009, New Trends in Data Warehousing and Data Analysis.

[5]  Manoj Kumar,et al.  Comparison of Data Warehouse Design Approaches from User Requirement to Conceptual Model: A Survey , 2011, 2011 International Conference on Communication Systems and Network Technologies.

[6]  Muhammad Younus Javed,et al.  Data Load Distribution by Semi Real Time Data Warehouse , 2010, 2010 Second International Conference on Computer and Network Technology.

[7]  A Min Tjoa,et al.  Zero-latency data warehousing (ZLDWH): the state-of-the-art and experimental implementation approaches , 2006, 2006 International Conference onResearch, Innovation and Vision for the Future.

[8]  Hao Ping,et al.  Research and design of the incremental updates of drug Data Warehouse , 2010, 2010 International Conference on Computer Application and System Modeling (ICCASM 2010).

[9]  Ge Yu,et al.  Study on Log-Based Change Data Capture and Handling Mechanism in Real-Time Data Warehouse , 2008, 2008 International Conference on Computer Science and Software Engineering.

[10]  Stefan Deßloch,et al.  Near Real-Time Data Warehousing Using State-of-the-Art ETL Tools , 2009, BIRTE.

[11]  Joann J. Ordille,et al.  Data integration: the teenage years , 2006, VLDB.

[12]  Prabhu Ram,et al.  Extracting delta for incremental data warehouse maintenance , 2000, Proceedings of 16th International Conference on Data Engineering (Cat. No.00CB37073).