DIAMOND: Correlation-Based Anomaly Monitoring Daemon for DIME

Distributed Interactive Multimedia Environments (DIMEs) show important dependency constraints between application and underlying system components over time. For example, the video frame rate and the underlying bandwidth usage have a strong performance dependency. Performance dependencies must also be considered among distributed components. These dependencies over a time-span form correlation relationships. Violations of such correlation relationships represent collective anomalies. Users and most specifically DIME application developers face problems of finding (detecting), localizing such anomalies, and adapting against them in real-time. Current practices are to collect joint application-system metadata characterizing behaviors of application and system components while a DIME session is running, and then analyze them offline. Our goal is to provide a framework, called DIAMOND, that allows for real-time and unobtrusive collection and organization of joint application-system metadata in order to assist in finding such correlation violations in the system. DIAMOND works in four steps: (a) real-time metadata collection, (b) metadata processing to allow efficient computation of correlation constraints, (c) metadata distribution for efficient clustering of distributed metadata, and (d) anomaly detection, localization, and evolution monitoring based on violations of correlation relationships. Our results from real implementations and simulations with Planet Lab traces show the effectiveness of DIAMOND in terms of network overhead and anomaly detection time.

[1]  Klara Nahrstedt,et al.  Quality of experience in distributed interactive multimedia environments: toward a theoretical framework , 2009, ACM Multimedia.

[2]  Indranil Gupta,et al.  Q-Tree: A Multi-Attribute Based Range Query Solution for Tele-immersive Framework , 2009, 2009 29th IEEE International Conference on Distributed Computing Systems.

[3]  Indranil Gupta,et al.  MON: management overlay networks for distributed systems , 2005, SOSP '05.

[4]  Mark Handley,et al.  A scalable content-addressable network , 2001, SIGCOMM '01.

[5]  Indranil Gupta,et al.  MON: On-Demand Overlays for Distributed System Management , 2005, WORLDS.

[6]  Branislav Rovan,et al.  On Partitioning Grids into Equal Parts , 1997, Comput. Artif. Intell..

[7]  Ting Wang.  TWStream: Finding Correlated Data Streams Under Time Warping , 2006, APWeb.

[8]  Walter Willinger,et al.  cSamp: A System for Network-Wide Flow Monitoring , 2008, NSDI.

[9]  VARUN CHANDOLA,et al.  Anomaly detection: A survey , 2009, CSUR.

[10]  Dennis Shasha,et al.  StatStream: Statistical Monitoring of Thousands of Data Streams in Real Time , 2002, VLDB.

[11]  Zhenhuan Gong,et al.  Self-correlating predictive information tracking for large-scale production systems , 2009, ICAC '09.

[12]  Christophe Diot,et al.  Design and evaluation of MiMaze a multi-player game on the Internet , 1998, Proceedings. IEEE International Conference on Multimedia Computing and Systems (Cat. No.98TB100241).

[13]  Indranil Gupta,et al.  Kelips: Building an Efficient and Stable P2P DHT through Increased Memory and Background Overhead , 2003, IPTPS.

[14]  Dina Q. Goldin,et al.  On Similarity Queries for Time-Series Data: Constraint Specification and Implementation , 1995, CP.

[15]  Klara Nahrstedt,et al.  Implementing a Distributed Tele-immersive System , 2008, 2008 Tenth IEEE International Symposium on Multimedia.

[16]  David R. Karger,et al.  Chord: A scalable peer-to-peer lookup service for internet applications , 2001, SIGCOMM '01.

[17]  Antony I. T. Rowstron,et al.  Pastry: Scalable, Decentralized Object Location, and Routing for Large-Scale Peer-to-Peer Systems , 2001, Middleware.

[18]  Thomas Malzbender,et al.  Understanding performance in coliseum, an immersive videoconferencing system , 2005, TOMCCAP.

[19]  Alberto O. Mendelzon,et al.  Querying Time Series Data Based on Similarity , 2000, IEEE Trans. Knowl. Data Eng..

[20]  Philip S. Yu,et al.  Mining Frequent Patterns in Data Streams at Multiple Time Granularities , 2002 .

[21]  Robert Morris,et al.  Chord: A scalable peer-to-peer lookup service for internet applications , 2001, SIGCOMM 2001.

[22]  Ruzena Bajcsy,et al.  TEEVE: the next generation architecture for tele-immersive environments , 2005, Seventh IEEE International Symposium on Multimedia (ISM'05).

[23]  Jiawei Han,et al.  Mining Approximate Top-K Subspace Anomalies in Multi-Dimensional Time-Series Data , 2007, VLDB.