Tracking transaction footprints for non-intrusive end-to-end monitoring

Existing transaction monitoring solutions are either platform-specific or rely on instrumentation techniques, which limit their applicability. Consequently, transaction monitoring in enterprise environments often involves the manual collation of information spread across a variety of infrastructure elements and applications, and is a time-consuming and labor-intensive task. To address this problem, we have developed an online, non-intrusive and platform-agnostic solution for transaction monitoring. The solution includes a transaction model discovery component that leverages historical system log files, containing transaction footprints and generates a model of the transaction in terms of valid sequence of steps that a transaction instance may execute and the expected footprint patterns at each step. The online monitoring system, in turn, takes in only (a) online system log files and (b) the transaction model, as inputs and generates a dynamic execution profile of ongoing transaction instances that allows their status to be tracked at individual and aggregate levels, even when transaction footprints do not necessarily carry correlating identifiers as those injected through instrumentation. In this paper, we describe the transaction model discovery and monitoring system including the architecture and algorithms, followed by results from an empirical study, ongoing work on run-time model validation and directions for future research.

[1]  Alexander L. Wolf,et al.  Discovering models of software processes from event-based data , 1998, TSEM.

[2]  Manish Gupta,et al.  Mining activity data for dynamic dependency discovery in e-business systems , 2004, IEEE Transactions on Network and Service Management.

[3]  Peer Hasselmeyer,et al.  Managing Dynamic Service Dependencies , 2001, DSOM.

[4]  Dimitrios Gunopulos,et al.  Mining Process Models from Workflow Logs , 1998, EDBT.

[5]  Joseph L. Hellerstein,et al.  Mining Event Data for Actionable Patterns , 2000, Int. CMG Conference.

[6]  Reinhold Kröger,et al.  A Generic Application-Oriented Performance Instrumentation for Multi-Tier Environments , 2007, 2007 10th IFIP/IEEE International Symposium on Integrated Network Management.

[7]  Bikram Sengupta,et al.  Data tagging architecture for system monitoring in dynamic environments , 2008, NOMS 2008 - 2008 IEEE Network Operations and Management Symposium.

[8]  Anima Anandkumar,et al.  Non-intrusive transaction monitoring using system logs , 2008, NOMS 2008 - 2008 IEEE Network Operations and Management Symposium.

[9]  Marcos K. Aguilera,et al.  Performance debugging for distributed systems of black boxes , 2003, SOSP '03.

[10]  Dimitris Karagiannis,et al.  An Inductive Approach to the Acquisition and Adaptation of Workflow Models , 1999 .

[11]  Anima Anandkumar,et al.  Tracking in a spaghetti bowl: monitoring transactions using footprints , 2008, SIGMETRICS '08.

[12]  Donna N. Dillenberger,et al.  Adaptive Algorithms for Managing a Distributed Data Processing Workload , 1997, IBM Syst. J..

[13]  Boudewijn F. van Dongen,et al.  Workflow mining: A survey of issues and approaches , 2003, Data Knowl. Eng..

[14]  Risto Vaarandi,et al.  A data clustering algorithm for mining patterns from event logs , 2003, Proceedings of the 3rd IEEE Workshop on IP Operations & Management (IPOM 2003) (IEEE Cat. No.03EX764).

[15]  David Sinreich,et al.  An architectural blueprint for autonomic computing , 2006 .

[16]  Wei Peng,et al.  Mining logs files for data-driven system management , 2005, SKDD.

[17]  Aaron B. Brown,et al.  An active approach to characterizing dynamic dependencies for problem determination in a distributed environment , 2001, 2001 IEEE/IFIP International Symposium on Integrated Network Management Proceedings. Integrated Network Management VII. Integrated Management Strategies for the New Millennium (Cat. No.01EX470).

[18]  Eric A. Brewer,et al.  Pinpoint: problem determination in large, dynamic Internet services , 2002, Proceedings International Conference on Dependable Systems and Networks.

[19]  Karsten Schwan,et al.  E2EProf: Automated End-to-End Performance Management for Enterprise Systems , 2007, 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN'07).