Loom: Complex large-scale visual insight for large hybrid IT infrastructure management

Abstract Interactive visual exploration techniques (IVET) such as those advocated by Shneiderman and extreme scale visual analytics have successfully increased our understanding of a variety of domains that produce huge amounts of complex data. In spite of their complexity, IT infrastructures have not benefited from the application of IVET techniques. Loom is inspired in IVET techniques and builds on them to tame increasing complexity in IT infrastructure management systems guaranteeing interactive response times and integrating key elements for IT management: Relationships between managed entities coming from different IT management subsystems, alerts and actions (or reconfigurations) of the IT setup. The Loom system builds on two main pillars: (1) a multiplex graph spanning data from different ITIMs; and (2) a novel visualisation arrangement: the Loom “Thread” visualisation model. We have tested this in a number of real-world applications, showing that Loom can handle million of entities without losing information, with minimum context switching, and offering better performance than other relational/graph-based systems. This ensures interactive response times (few seconds as 90th percentile). The value of the “Thread” visualisation model is shown in a qualitative analysis of users’ experiences with Loom.

[1]  Christophe Hurter,et al.  Attribute-driven edge bundling for general graphs with applications in trail analysis , 2015, 2015 IEEE Pacific Visualization Symposium (PacificVis).

[2]  Paul A. Cairns,et al.  Beyond guidelines: what can we learn from the visual information seeking mantra? , 2005, Ninth International Conference on Information Visualisation (IV'05).

[3]  Yixin Diao,et al.  IT service management automation - A hybrid methodology to integrate and orchestrate collaborative human centric and automation centric workflows , 2007, IEEE International Conference on Services Computing (SCC 2007).

[4]  Yehuda Koren,et al.  Topological fisheye views for visualizing large graphs , 2004, IEEE Transactions on Visualization and Computer Graphics.

[5]  James Abello,et al.  ASK-GraphView: A Large Scale Graph Visualization System , 2006, IEEE Transactions on Visualization and Computer Graphics.

[6]  Martin Wattenberg,et al.  Visual exploration of multivariate graphs , 2006, CHI.

[7]  Ben Shneiderman,et al.  The eyes have it: a task by data type taxonomy for information visualizations , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[8]  Jean-Daniel Fekete,et al.  Hierarchical Aggregation for Information Visualization: Overview, Techniques, and Design Guidelines , 2010, IEEE Transactions on Visualization and Computer Graphics.

[9]  Alexandr Andoni,et al.  Interacting with Large Distributed Datasets Using Sketch , 2016, EGPGV@EuroVis.

[10]  Jean-Daniel Fekete.  ProgressiVis: a Toolkit for Steerable Progressive Analytics and Visualization , 2015 .

[11]  Divesh Srivastava,et al.  Incremental Record Linkage , 2014, Proc. VLDB Endow..

[12]  Michael J. McGuffin,et al.  VisReduce: Fast and responsive incremental information visualization of large datasets , 2013, 2013 IEEE International Conference on Big Data.

[13]  Jarke J. van Wijk,et al.  Multivariate Network Exploration and Presentation: From Detail to Overview via Selections and Aggregations , 2014, IEEE Transactions on Visualization and Computer Graphics.

[14]  Jez Humble,et al.  Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation , 2010 .

[15]  Deep Ganguli,et al.  Druid: a real-time analytical data store , 2014, SIGMOD Conference.

[16]  Eugene Wu sirrice.  The Case for Data Visualization Management Systems [ Vision Paper ] , 2014 .

[17]  Michael Stonebraker,et al.  Dynamic reduction of query result sets for interactive visualizaton , 2013, 2013 IEEE International Conference on Big Data.

[18]  Ben Shneiderman,et al.  Motif simplification: improving network visualization readability with fan, connector, and clique glyphs , 2013, CHI.

[19]  Thomas W. Crockett,et al.  An Introduction to Parallel Rendering , 1997, Parallel Comput..

[20]  Ken Perlin,et al.  Pad: an alternative approach to the computer interface , 1993, SIGGRAPH.

[21]  Pat Hanrahan,et al.  Polaris: a system for query, analysis and visualization of multi-dimensional relational databases , 2000, IEEE Symposium on Information Visualization 2000. INFOVIS 2000. Proceedings.

[22]  Lei Shi,et al.  Scalable network traffic visualization using compressed graphs , 2013, 2013 IEEE International Conference on Big Data.

[23]  Danyel Fisher,et al.  Big data exploration requires collaboration between visualization and data infrastructures , 2016, HILDA '16.

[24]  Pak Chung Wong,et al.  Top Ten Interaction Challenges in Extreme-Scale Visual Analytics , 2012, Expanding the Frontiers of Visual Analytics and Visualization.

[25]  Abhishek Verma,et al.  Large-scale cluster management at Google with Borg , 2015, EuroSys.

[26]  Andrey Gubarev,et al.  Dremel : Interactive Analysis of Web-Scale Datasets , 2011 .

[27]  Jeffrey Heer,et al.  imMens: Real‐time Visual Querying of Big Data , 2013, Comput. Graph. Forum.

[28]  Lei Shi,et al.  Visualizing large graphs , 2015 .

[29]  Walter Didimo,et al.  Kojaph: Visual Definition and Exploration of Patterns in Graph Databases , 2015, GD.

[30]  Scott Shenker,et al.  Spark: Cluster Computing with Working Sets , 2010, HotCloud.

[31]  Minsuk Kahng,et al.  Scalable graph exploration and visualization: Sensemaking challenges and opportunities , 2015, 2015 International Conference on Big Data and Smart Computing (BIGCOMP).

[32]  John Allen,et al.  Scuba: Diving into Data at Facebook , 2013, Proc. VLDB Endow..

[33]  Jeffrey Heer,et al.  Enterprise Data Analysis and Visualization: An Interview Study , 2012, IEEE Transactions on Visualization and Computer Graphics.

[34]  Robert B. Ross,et al.  The Top 10 Challenges in Extreme-Scale Visual Analytics , 2012, IEEE Computer Graphics and Applications.

[35]  Aba-Sah Dadzie,et al.  Approaches to visualising Linked Data: A survey , 2011, Semantic Web.

[36]  Yue Zhao,et al.  Hierarchical Focus+Context Heterogeneous Network Visualization , 2014, 2014 IEEE Pacific Visualization Symposium.

[37]  Valerio Pascucci,et al.  Parallel visualization on large clusters using MapReduce , 2011, 2011 IEEE Symposium on Large Data Analysis and Visualization.

[38]  Alexander Hall,et al.  Processing a Trillion Cells per Mouse Click , 2012, Proc. VLDB Endow..

[39]  Alvaro Graves,et al.  Techniques to reduce cluttering of RDF visualizations , 2015, Future Gener. Comput. Syst..

[40]  Alexandru Telea,et al.  CUBu: Universal Real-Time Bundling for Large Graphs , 2016, IEEE Transactions on Visualization and Computer Graphics.

[41]  Kristin A. Cook,et al.  Illuminating the Path: The Research and Development Agenda for Visual Analytics , 2005 .

[42]  Kang Zhang,et al.  Visualizing big graphs with labels through edge bundling , 2016, 2016 IEEE International Conference on Big Data Analysis (ICBDA).

[43]  Timos K. Sellis,et al.  Exploration and Visualization in the Web of Big Linked Data: A Survey of the State of the Art , 2016, EDBT/ICDT Workshops.