Leveraging Business Process Models for ETL Design

As Business Intelligence evolves from off-line strategic decision making to on-line operational decision making, the design of the back-end Extract-Transform-Load (ETL) processes is becoming even more complex. Many challenges arise in this new context like their optimization and modeling. In this paper, we focus on the disconnection between the IT-level view of the enterprise presented by ETL processes and the business view of the enterprise required by managers and analysts. We propose the use of business process models for a conceptual view of ETL. We show how to link this conceptual view to existing business processes and how to translate from this conceptual view to a logical ETL view that can be optimized. Thus, we link the ETL processes back to their underlying business processes and so enable not only a business view of the ETL, but also a near real-time view of the entire enterprise.

[1]  Sunita Sarawagi,et al.  Modeling multidimensional databases , 1997, Proceedings 13th International Conference on Data Engineering.

[2]  Panos Vassiliadis,et al.  Conceptual modeling for ETL processes , 2002, DOLAP '02.

[3]  W.M.P. van der Aalst,et al.  Pattern and XPDL: a critical evaluation of the XML process definition language , 2003 .

[4]  Hongjun Lu,et al.  Conceptual Modeling – ER 2004 , 2004, Lecture Notes in Computer Science.

[5]  Panos Vassiliadis,et al.  Data Mapping Diagrams for Data Warehouse Design with UML , 2004, ER.

[6]  Panos Vassiliadis,et al.  A generic and customizable framework for the design of ETL scenarios , 2005, Inf. Syst..

[7]  Mario Piattini,et al.  Applying MDA to the development of data warehouses , 2005, DOLAP '05.

[8]  Timos K. Sellis,et al.  Optimizing ETL processes in data warehouses , 2005, 21st International Conference on Data Engineering (ICDE'05).

[9]  Jianwen Su,et al.  Towards Formal Analysis of Artifact-Centric Business Process Models , 2007, BPM.

[10]  Wolfgang Lehner,et al.  BPEL-DT - Data-aware Extension of BPEL to Support Data-Intensive Service Applications , 2007, WEWST.

[11]  Panos Vassiliadis,et al.  A method for the mapping of conceptual designs to logical blueprints for ETL processes , 2008, Decis. Support Syst..

[12]  Esteban Zimányi,et al.  Defining ETL worfklows using BPMN and BPEL , 2009, DOLAP.

[13]  Kevin Wilkinson,et al.  Data integration flows for business intelligence , 2009, EDBT '09.

[14]  Kevin Wilkinson,et al.  QoX-driven ETL design: reducing the cost of ETL consulting engagements , 2009, SIGMOD Conference.

[15]  Kevin Wilkinson,et al.  Business Processes Meet Operational Business Intelligence , 2009, IEEE Data Eng. Bull..

[16]  Jose-Norberto Mazón,et al.  Automatic generation of ETL processes from conceptual models , 2009, DOLAP.

[17]  Kevin Wilkinson,et al.  Optimizing ETL workflows for fault-tolerance , 2010, 2010 IEEE 26th International Conference on Data Engineering (ICDE 2010).