MIRIA: A Mixed Reality Toolkit for the In-Situ Visualization and Analysis of Spatio-Temporal Interaction Data

In this paper, we present MIRIA, a Mixed Reality Interaction Analysis toolkit designed to support the in-situ visual analysis of user interaction in mixed reality and multi-display environments. So far, there are few options to effectively explore and analyze interaction patterns in such novel computing systems. With MIRIA, we address this gap by supporting the analysis of user movement, spatial interaction, and event data by multiple, co-located users directly in the original environment. Based on our own experiences and an analysis of the typical data, tasks, and visualizations used in existing approaches, we identify requirements for our system. We report on the design and prototypical implementation of MIRIA, which is informed by these requirements and offers various visualizations such as 3D movement trajectories, position heatmaps, and scatterplots. To demonstrate the value of MIRIA for real-world analysis tasks, we conducted expert feedback sessions using several use cases with authentic study data.

[1]  Raimund Dachselt,et al.  GIAnT: Visualizing Group Interaction at Large Wall Displays , 2017, CHI.

[2]  Mateu Sbert,et al.  A New Scheme for Trajectory Visualization , 2014, 2014 18th International Conference on Information Visualisation.

[3]  Mishal Dholakia,et al.  Immersive Insights: A Hybrid Analytics System forCollaborative Exploratory Data Analysis , 2019, VRST.

[4]  Maximilian Speicher,et al.  MRAT: The Mixed Reality Analytics Toolkit , 2020, CHI.

[5]  Thies Pfeiffer.  Measuring and visualizing attention in space with 3D attention volumes , 2012, ETRA '12.

[6]  Nicolai Marquardt,et al.  EagleView: A Video Analysis Tool for Visualising and Querying Spatial Interactions of People and Devices , 2018, ISS.

[7]  David Ott,et al.  Kinect analysis: a system for recording, analysing and sharing multimodal interaction elicitation studies , 2015, EICS.

[8]  Brian P. Bailey,et al.  VICPAM: A Visualization Tool for Examining Interaction Data in Multiple Display Environments , 2011, HCI.

[9]  Deb Roy,et al.  An immersive system for browsing and visualizing surveillance video , 2010, ACM Multimedia.

[10]  Daniel A. Keim,et al.  Visual Analytics of Movement , 2013, Springer Berlin Heidelberg.

[11]  Carla Maria Dal Sasso Freitas,et al.  VirtualDesk: A Comfortable and Efficient Immersive Information Visualization Approach , 2018, Comput. Graph. Forum.

[12]  Raimund Dachselt,et al.  Personal Augmented Reality for Information Visualization on Large Interactive Displays , 2020, IEEE Transactions on Visualization and Computer Graphics.

[13]  Arnaud Prouzeau,et al.  Personal+Context navigation: combining AR and shared displays in network path-following , 2020, Graphics Interface.

[14]  Tovi Grossman,et al.  Patina: dynamic heatmaps for visualizing application usage , 2013, CHI.

[15]  Keith Cheverst,et al.  3D Space-Time Visualization of Player Behaviour in Pervasive Location-Based Games , 2008, Int. J. Comput. Games Technol..

[16]  Raimund Dachselt,et al.  Investigating Smartphone-based Pan and Zoom in 3D Data Spaces in Augmented Reality , 2019, MobileHCI.

[17]  Wendy E. Mackay,et al.  Effects of display size and navigation type on a classification task , 2014, CHI.

[18]  M. Sheelagh T. Carpendale,et al.  Territoriality in collaborative tabletop workspaces , 2004, CSCW.

[19]  David Lindlbauer,et al.  HeatSpace: Automatic Placement of Displays by Empirical Analysis of User Behavior , 2017, UIST.

[20]  Alessandro Canossa,et al.  Analyzing spatial user behavior in computer games using geographic information systems , 2009, MindTrek '09.

[21]  Christophe Hurter,et al.  IATK: An Immersive Analytics Toolkit , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[22]  Niklas Elmqvist,et al.  There Is No Spoon: Evaluating Performance, Space Use, and Presence with Expert Domain Users in Immersive Analytics , 2019, IEEE Transactions on Visualization and Computer Graphics.

[23]  Anastasia Bezerianos,et al.  Collaborative Immersive Analytics , 2018, Immersive Analytics.

[24]  Tovi Grossman,et al.  BISHARE: Exploring Bidirectional Interactions Between Smartphones and Head-Mounted Augmented Reality , 2020, CHI.

[25]  Fabrice Matulic,et al.  BodyLenses: Embodied Magic Lenses and Personal Territories for Wall Displays , 2015, ITS.

[26]  Chris North,et al.  Realizing embodied interaction for visual analytics through large displays , 2007, Comput. Graph..

[27]  Keita Higuchi,et al.  Browsing Group First-Person Videos with 3D Visualization , 2018, ISS.

[28]  Halldór Janetzko,et al.  SimpliFly: A Methodology for Simplification and Thematic Enhancement of Trajectories , 2015, IEEE Transactions on Visualization and Computer Graphics.

[29]  Mikkel Rønne Jakobsen,et al.  Negotiating for Space?: Collaborative Work Using a Wall Display with Mouse and Touch Input , 2016, CHI.

[30]  Maneesh Agrawala,et al.  Visualizing competitive behaviors in multi-user virtual environments , 2004, IEEE Visualization 2004.

[31]  Romain Vuillemot,et al.  ReViVD: Exploration and Filtering of Trajectories in an Immersive Environment using 3D Shapes , 2020, 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[32]  Sven Wachsmuth,et al.  Integrating PAMOCAT in the research cycle: linking motion capturing and conversation analysis , 2012, ICMI '12.

[33]  Olivier Chapuis,et al.  Leveraging Body Interactions to Support Immersive Analytics , 2019, CHI 2019.

[34]  Chris North,et al.  Immersive Analytics: Theory and Research Agenda , 2019, Front. Robot. AI.

[35]  Lennart E. Nacke,et al.  Unified visualization of quantitative and qualitative playtesting data , 2014, CHI Extended Abstracts.

[36]  Ricardo Langner,et al.  Investigating the Use of Spatial Interaction for 3D Data Visualization on Mobile Devices , 2017, ISS.

[37]  Stefan Gumhold,et al.  Partial Matching of Trajectories with Particle Orientation for Exploratory Trajectory Visualization , 2020, VMV.

[38]  Mikkel Rønne Jakobsen,et al.  Is Moving Improving?: Some Effects of Locomotion in Wall-Display Interaction , 2015, CHI.

[39]  Wolfgang Stuerzlinger,et al.  Evaluating an Immersive Space-Time Cube Geovisualization for Intuitive Trajectory Data Exploration , 2019, IEEE Transactions on Visualization and Computer Graphics.

[40]  Chris North,et al.  Interaction for Immersive Analytics , 2018, Immersive Analytics.

[41]  Karthik Ramani,et al.  GestureAnalyzer: visual analytics for pattern analysis of mid-air hand gestures , 2014, SUI.

[42]  Raimund Dachselt,et al.  Miners: Communication and Awareness in Collaborative Gaming at an Interactive Display Wall , 2016, ISS.

[43]  Ke Huo,et al.  GhostAR: A Time-space Editor for Embodied Authoring of Human-Robot Collaborative Task with Augmented Reality , 2019, UIST.

[44]  Ross T. Smith,et al.  Situated Analytics , 2018, Immersive Analytics.

[45]  Thierry Baccino,et al.  UX Heatmaps: Mapping User Experience on Visual Interfaces , 2016, CHI.

[46]  Tsvi Kuflik,et al.  Analyzing Museum Visitors' Behavior Patterns , 2007, User Modeling.

[47]  Fabian Beck,et al.  A Design and Application Space for Visualizing User Sessions of Virtual and Mixed Reality Environments , 2020, VMV.

[48]  Günter Wallner,et al.  Aggregated Visualization of Playtesting Data , 2019, CHI.

[49]  Marc Alexa,et al.  OptiSpace: Automated Placement of Interactive 3D Projection Mapping Content , 2018, CHI.

[50]  Bruce H. Thomas,et al.  ImAxes: Immersive Axes as Embodied Affordances for Interactive Multivariate Data Visualisation , 2017, UIST.

[51]  Günter Wallner,et al.  A spatiotemporal visualization approach for the analysis of gameplay data , 2012, CHI.

[52]  Katy Börner,et al.  Social Diffusion Patterns in Three-Dimensional Virtual Worlds , 2003, Inf. Vis..

[53]  M. Sheelagh T. Carpendale,et al.  VisTACO: visualizing tabletop collaboration , 2010, ITS '10.

[54]  Michael Lankes,et al.  See, Feel, Move: Player Behaviour Analysis through Combined Visualization of Gaze, Emotions, and Movement , 2020, CHI.

[55]  Jarke J. van Wijk,et al.  Interactive visualization of multivariate trajectory data with density maps , 2011, 2011 IEEE Pacific Visualization Symposium.

[56]  Tsvi Kuflik,et al.  Visualizing Proximity‐Based Spatiotemporal Behavior of Museum Visitors using Tangram Diagrams , 2014, Comput. Graph. Forum.

[57]  Matthias Klapperstück,et al.  Immersive Analytics , 2015, 2015 Big Data Visual Analytics (BDVA).

[58]  Wei Chen,et al.  ForVizor: Visualizing Spatio-Temporal Team Formations in Soccer , 2019, IEEE Transactions on Visualization and Computer Graphics.

[59]  Ricardo Langner,et al.  CoFind: a browser plugin for investigating co-located collaborative web search , 2020, Mensch & Computer.

[60]  Anthony Tang,et al.  EXCITE: EXploring Collaborative Interaction in Tracked Environments , 2015, INTERACT.

[61]  Christopher D. Shaw,et al.  Visualizing and understanding players' behavior in video games: discovering patterns and supporting aggregation and comparison , 2011, Sandbox '11.

[62]  Panagiotis D. Ritsos,et al.  VRIA: A Web-Based Framework for Creating Immersive Analytics Experiences , 2020, IEEE Transactions on Visualization and Computer Graphics.

[63]  Christophe Hurter,et al.  FiberClay: Sculpting Three Dimensional Trajectories to Reveal Structural Insights , 2019, IEEE Transactions on Visualization and Computer Graphics.

[64]  Won-Ki Jeong,et al.  DXR: A Toolkit for Building Immersive Data Visualizations , 2019, IEEE Transactions on Visualization and Computer Graphics.

[65]  Nadir Weibel,et al.  ChronoViz: a system for supporting navigation of time-coded data , 2011, CHI Extended Abstracts.

[66]  Ying Zhang,et al.  Fly with the flock: immersive solutions for animal movement visualization and analytics , 2019, Journal of the Royal Society Interface.

[67]  Yalong Yang,et al.  Origin-Destination Flow Maps in Immersive Environments , 2019, IEEE Transactions on Visualization and Computer Graphics.

[68]  Adrien Fonnet,et al.  Survey of Immersive Analytics , 2021, IEEE Transactions on Visualization and Computer Graphics.

[69]  Luca Chittaro,et al.  VU-Flow: A Visualization Tool for Analyzing Navigation in Virtual Environments , 2006, IEEE Transactions on Visualization and Computer Graphics.

[70]  Ricardo Langner,et al.  Multiple Coordinated Views at Large Displays for Multiple Users: Empirical Findings on User Behavior, Movements, and Distances , 2019, IEEE Transactions on Visualization and Computer Graphics.

[71]  Kasper Hornbæk,et al.  Who Put That There? Temporal Navigation of Spatial Recordings by Direct Manipulation , 2020, CHI.

[72]  Heidrun Schumann,et al.  Stacking-Based Visualization of Trajectory Attribute Data , 2012, IEEE Transactions on Visualization and Computer Graphics.

[73]  Kwan-Liu Ma,et al.  Collaborative Visual Analysis with Multi-level Information Sharing Using a Wall-Size Display and See-Through HMDs , 2019, 2019 IEEE Pacific Visualization Symposium (PacificVis).

[74]  Aidong Lu,et al.  Improving Information Sharing and Collaborative Analysis for Remote GeoSpatial Visualization Using Mixed Reality , 2019, 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[75]  Alessio Del Bue,et al.  Social interaction discovery by statistical analysis of F-formations , 2011, BMVC.

[76]  Tobias Schreck,et al.  Immersive analysis of user motion in VR applications , 2020, The Visual Computer.

[77]  Arie E. Kaufman,et al.  VEEVVIE: Visual Explorer for Empirical Visualization, VR and Interaction Experiments , 2016, IEEE Transactions on Visualization and Computer Graphics.

[78]  Magdalena Balazinska,et al.  Specification and Verification of Complex Location Events with Panoramic , 2010, Pervasive.

[79]  Raimund Dachselt,et al.  Reality Graph Visualizations Investigation of Visual Styles in 3D Node-Link Diagrams , 2019 .

[80]  Harald Reiterer,et al.  Clusters, Trends, and Outliers: How Immersive Technologies Can Facilitate the Collaborative Analysis of Multidimensional Data , 2018, CHI.

[81]  Gennady L. Andrienko,et al.  Analysis of Flight Variability: a Systematic Approach , 2019, IEEE Transactions on Visualization and Computer Graphics.