Measuring precision of modeled behavior

Conformance checking techniques compare observed behavior (i.e., event logs) with modeled behavior for a variety of reasons. For example, discrepancies between a normative process model and recorded behavior may point to fraud or inefficiencies. The resulting diagnostics can be used for auditing and compliance management. Conformance checking can also be used to judge a process model automatically discovered from an event log. Models discovered using different process discovery techniques need to be compared objectively. These examples illustrate just a few of the many use cases for aligning observed and modeled behavior. Thus far, most conformance checking techniques focused on replay fitness, i.e., the ability to reproduce the event log. However, it is easy to construct models that allow for lots of behavior (including the observed behavior) without being precise. In this paper, we propose a method to measure precision of process models, given their event logs by first aligning the logs to the models. This way, the measurement is not sensitive to non-fitting executions and more accurate values can be obtained for non-fitting logs. Furthermore, we introduce several variants of the technique to deal better with incomplete logs and reduce possible bias due to behavioral property of process models. The approach has been implemented in the ProM 6 framework and tested against both artificial and real-life cases. Experiments show that the approach is robust to noise and applicable to handle logs and models of real-life complexity.

[1]  Boudewijn F. van Dongen,et al.  Alignment Based Precision Checking , 2012, Business Process Management Workshops.

[2]  Boudewijn F. van Dongen,et al.  Conformance Checking Using Cost-Based Fitness Analysis , 2011, 2011 IEEE 15th International Enterprise Distributed Object Computing Conference.

[3]  Karsten Wolf.  Stubborn Sets for Standard Properties , 1999, ICATPN.

[4]  Nicola Zannone,et al.  Measuring Privacy Compliance with Process Specifications , 2011, 2011 Third International Workshop on Security Measurements and Metrics.

[5]  Tharam S. Dillon,et al.  On the Move to Meaningful Internet Systems: OTM 2011 Workshops - Confederated International Workshops and Posters: EI2N+NSF ICE, ICSP+INBAST, ISDE, ORM, OTMA, SWWS+MONET+SeDeS, and VADER 2011, Hersonissos, Crete, Greece, October 17-21, 2011. Proceedings , 2011, OTM Workshops.

[6]  Jan Mendling,et al.  Process Compliance Measurement Based on Behavioural Profiles , 2010, CAiSE.

[7]  Peter S. Pande,et al.  The Six Sigma Way: How GE, Motorola, and Other Top Companies are Honing Their Performance , 2000 .

[8]  Bart Baesens,et al.  A robust F-measure for evaluating discovered process models , 2011, 2011 IEEE Symposium on Computational Intelligence and Data Mining (CIDM).

[9]  Wil M. P. van der Aalst,et al.  Process Mining - Discovery, Conformance and Enhancement of Business Processes , 2011 .

[10]  Wil M. P. van der Aalst,et al.  Process Mining Applied to the Test Process of Wafer Scanners in ASML , 2009, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[11]  Bart Baesens,et al.  Improved Artificial Negative Event Generation to Enhance Process Event Logs , 2012, CAiSE.

[12]  Boudewijn F. van Dongen,et al.  Towards Cross-Organizational Process Mining in Collections of Process Models and Their Executions , 2011, Business Process Management Workshops.

[13]  van der Wmp Wil Aalst,et al.  Memory-efficient alignment of observed and modeled behavior , 2013 .

[14]  Tadao Murata,et al.  Petri nets: Properties, analysis and applications , 1989, Proc. IEEE.

[15]  Tharam S. Dillon,et al.  On the Move to Meaningful Internet Systems, OTM 2010 , 2010, Lecture Notes in Computer Science.

[16]  Jorge Munoz-Gama,et al.  A GENERAL FRAMEWORK FOR PRECISION CHECKING , 2012 .

[17]  Serge Haddad,et al.  Application and Theory of Petri Nets , 2012, Lecture Notes in Computer Science.

[18]  Wil M. P. van der Aalst,et al.  Conformance checking of processes based on monitoring real behavior , 2008, Inf. Syst..

[19]  Wil M. P. van der Aalst,et al.  Modern Business Process Automation: YAWL and its Support Environment , 2009 .

[20]  Remco M. Dijkman,et al.  Semantics and analysis of business process models in BPMN , 2008, Inf. Softw. Technol..

[21]  Jorge S. Cardoso,et al.  Measuring the Compliance of Processes with Reference Models , 2009, OTM Conferences.

[22]  Bart Baesens,et al.  A comprehensive benchmarking framework (CoBeFra) for conformance analysis between procedural process models and event logs in ProM , 2013, 2013 IEEE Symposium on Computational Intelligence and Data Mining (CIDM).

[23]  Boudewijn F. van Dongen,et al.  Cost-Based Fitness in Conformance Checking , 2011, 2011 Eleventh International Conference on Application of Concurrency to System Design.

[24]  Boudewijn F. van Dongen,et al.  On the Role of Fitness, Precision, Generalization and Simplicity in Process Discovery , 2012, OTM Conferences.

[25]  Nicola Zannone,et al.  Purpose Control: Did You Process the Data for the Intended Purpose? , 2011, Secure Data Management.

[26]  Boudewijn F. van Dongen,et al.  Process mining: a two-step approach to balance between underfitting and overfitting , 2008, Software & Systems Modeling.

[27]  Luigi Pontieri,et al.  Discovering expressive process models by clustering log traces , 2006, IEEE Transactions on Knowledge and Data Engineering.

[28]  Boudewijn F. van Dongen,et al.  Causal Nets: A Modeling Language Tailored towards Process Discovery , 2011, CONCUR.

[29]  Lars Michael Kristensen,et al.  Question-guided stubborn set methods for state properties , 2006, Formal Methods Syst. Des..

[30]  L. J. Porter,et al.  Total quality management—the critical success factors , 1993 .

[31]  Alexander L. Wolf,et al.  Software process validation: quantitatively measuring the correspondence of a process to a model , 1999, TSEM.

[32]  Bart Baesens,et al.  A Critical Evaluation Study of Model-Log Metrics in Process Discovery , 2010, Business Process Management Workshops.

[33]  Wil M. P. van der Aalst.  Business Process Configuration in the Cloud: How to Support and Analyze Multi-tenant Processes? , 2011, 2011 IEEE Ninth European Conference on Web Services.

[34]  Boudewijn F. van Dongen,et al.  Replaying history on process models for conformance checking and performance analysis , 2012, WIREs Data Mining Knowl. Discov..

[35]  Kees M. van Hee,et al.  Workflow Management: Models, Methods, and Systems , 2002, Cooperative information systems.

[36]  Josep Carmona,et al.  A Fresh Look at Precision in Process Conformance , 2010, BPM.

[37]  Wil vanderAalst,et al.  Workflow Management: Models, Methods, and Systems , 2004 .

[38]  Jorge Munoz-Gama,et al.  Enhancing precision in Process Conformance: Stability, confidence and severity , 2011, 2011 IEEE Symposium on Computational Intelligence and Data Mining (CIDM).