AvatAR: An Immersive Analysis Environment for Human Motion Data Combining Interactive 3D Avatars and Trajectories

Analysis of human motion data can reveal valuable insights about the utilization of space and interaction of humans with their environment. To support this, we present AvatAR, an immersive analysis environment for the in-situ visualization of human motion data, that combines 3D trajectories with virtual avatars showing people’s detailed movement and posture. Additionally, we describe how visualizations can be embedded directly into the environment, showing what a person looked at or what surfaces they touched, and how the avatar’s body parts can be used to access and manipulate those visualizations. AvatAR combines an AR HMD with a tablet to provide both mid-air and touch interaction for system control, as well as an additional overview device to help users navigate the environment. We implemented a prototype and present several scenarios to show that AvatAR can enhance the analysis of human motion data by making data not only explorable, but experienceable.

[1]  Kasper Hornbæk,et al.  Who Put That There? Temporal Navigation of Spatial Recordings by Direct Manipulation , 2020, CHI.

[2]  M. Sheelagh T. Carpendale,et al.  Immersive Analytics: Exploring Future Interaction and Visualization Technologies for Data Analytics , 2016, ISS.

[3]  Tsvi Kuflik,et al.  Visualizing museum visitors’ behavior: Where do they go and what do they do there? , 2017, Personal and Ubiquitous Computing.

[4]  Jürgen Beyerer,et al.  Toward Mass Video Data Analysis: Interactive and Immersive 4D Scene Reconstruction , 2020, Sensors.

[5]  Panagiotis D. Ritsos,et al.  VRIA: A Web-Based Framework for Creating Immersive Analytics Experiences , 2020, IEEE Transactions on Visualization and Computer Graphics.

[6]  Raimund Dachselt,et al.  Personal Augmented Reality for Information Visualization on Large Interactive Displays , 2020, IEEE Transactions on Visualization and Computer Graphics.

[7]  Heidrun Schumann,et al.  Visual Methods for Analyzing Time-Oriented Data , 2008, IEEE Transactions on Visualization and Computer Graphics.

[8]  Katy Börner,et al.  Social Diffusion Patterns in Three-Dimensional Virtual Worlds , 2003, Inf. Vis..

[9]  Wolfgang Stuerzlinger,et al.  Evaluating an Immersive Space-Time Cube Geovisualization for Intuitive Trajectory Data Exploration , 2019, IEEE Transactions on Visualization and Computer Graphics.

[10]  Nicolai Marquardt,et al.  EagleView , 2018, Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces.

[11]  Tobias Isenberg,et al.  Supporting Volumetric Data Visualization and Analysis by Combining Augmented Reality Visuals with Multi-Touch Input , 2019, EuroVis.

[12]  Sabine Coquillart,et al.  Editorial , 1999, Comput. Graph. Forum.

[13]  Ke Huo,et al.  GhostAR: A Time-space Editor for Embodied Authoring of Human-Robot Collaborative Task with Augmented Reality , 2019, UIST.

[14]  Heidrun Schumann,et al.  Stacking-Based Visualization of Trajectory Attribute Data , 2012, IEEE Transactions on Visualization and Computer Graphics.

[15]  Nicolai Marquardt,et al.  EagleView: A Video Analysis Tool for Visualising and Querying Spatial Interactions of People and Devices , 2018, ISS.

[16]  Luciana Porcher Nedel,et al.  Immersive Visualization of Abstract Information: An Evaluation on Dimensionally-Reduced Data Scatterplots , 2018, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[17]  D. Keim,et al.  Dynamic Visual Abstraction of Soccer Movement , 2017, Computer graphics forum (Print).

[18]  Maneesh Agrawala,et al.  Visualizing competitive behaviors in multi-user virtual environments , 2004, IEEE Visualization 2004.

[19]  Atsushi Nakazawa,et al.  Analysis and synthesis of human dance motions , 2003, Proceedings of IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, MFI2003..

[20]  Stefan Buschmann,et al.  Animated visualization of spatial–temporal trajectory data for air-traffic analysis , 2016, The Visual Computer.

[21]  Anthony Tang,et al.  EXCITE: EXploring Collaborative Interaction in Tracked Environments , 2015, INTERACT.

[22]  Johannes Zagermann,et al.  STREAM: Exploring the Combination of Spatially-Aware Tablets with Augmented Reality Head-Mounted Displays for Immersive Analytics , 2021, CHI.

[23]  Bernd Hamann,et al.  Involve Me and I Will Understand!-Abstract Data Visualization in Immersive Environments , 2011, ISVC.

[24]  Ricardo Langner,et al.  MARVIS: Combining Mobile Devices and Augmented Reality for Visual Data Analysis , 2021, CHI.

[25]  Minh Vo,et al.  Long-term Human Motion Prediction with Scene Context , 2020, ECCV.

[26]  Elie Azar,et al.  Agent-Based Modeling of Occupants and Their Impact on Energy Use in Commercial Buildings , 2012, J. Comput. Civ. Eng..

[27]  Mishal Dholakia,et al.  Immersive Insights: A Hybrid Analytics System forCollaborative Exploratory Data Analysis , 2019, VRST.

[28]  Yubo Tao,et al.  Visual analytics of urban transportation from a bike-sharing and taxi perspective , 2020, J. Vis..

[29]  Timothy M. Kowalewski,et al.  Trajectory Mapper: Interactive Widgets and Artist-Designed Encodings for Visualizing Multivariate Trajectory Data , 2017, EuroVis.

[30]  Yaser Sheikh,et al.  Hand Keypoint Detection in Single Images Using Multiview Bootstrapping , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Gennady L. Andrienko,et al.  Clustering Trajectories by Relevant Parts for Air Traffic Analysis , 2018, IEEE Transactions on Visualization and Computer Graphics.

[32]  Tamara Munzner,et al.  Ocupado: Visualizing Location‐Based Counts Over Time Across Buildings , 2020, Comput. Graph. Forum.

[33]  Rubaiat Habib Kazi,et al.  SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ , 2018, CHI.

[34]  Tobias Isenberg,et al.  Immersive Analytics: An Introduction , 2018, Immersive Analytics.

[35]  Pan Zhang,et al.  Semantic human activity annotation tool using skeletonized surveillance videos , 2019, UbiComp/ISWC Adjunct.

[36]  Dieter Schmalstieg,et al.  Bridging multiple user interface dimensions with augmented reality , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[37]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Gennady L. Andrienko,et al.  Exploratory spatio-temporal visualization: an analytical review , 2003, J. Vis. Lang. Comput..

[39]  Tobias Schreck,et al.  Immersive analysis of user motion in VR applications , 2020, The Visual Computer.

[40]  Denis Kalkofen,et al.  Perspective Matters: Design Implications for Motion Guidance in Mixed Reality , 2020, 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[41]  Maximilian Speicher,et al.  MRAT: The Mixed Reality Analytics Toolkit , 2020, CHI.

[42]  Ricardo Langner,et al.  Investigating the Use of Spatial Interaction for 3D Data Visualization on Mobile Devices , 2017, ISS.

[43]  Mishal Dholakia,et al.  Dataspace: A Reconfigurable Hybrid Reality Environment for Collaborative Information Analysis , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[44]  Christopher D. Shaw,et al.  Visualizing and understanding players' behavior in video games: discovering patterns and supporting aggregation and comparison , 2011, Sandbox '11.

[45]  Takeo Kanade,et al.  Panoptic Studio: A Massively Multiview System for Social Interaction Capture , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[46]  Bodo Rosenhahn,et al.  Supplementary Material to: Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera , 2018 .

[47]  Raimund Dachselt,et al.  MIRIA: A Mixed Reality Toolkit for the In-Situ Visualization and Analysis of Spatio-Temporal Interaction Data , 2021, CHI.

[48]  Ross T. Smith,et al.  Temporal-Geospatial Cooperative Visual Analysis , 2016, 2016 Big Data Visual Analytics (BDVA).

[49]  Michael Lankes,et al.  See, Feel, Move: Player Behaviour Analysis through Combined Visualization of Gaze, Emotions, and Movement , 2020, CHI.

[50]  Gennady L. Andrienko,et al.  Visual analytics of movement: An overview of methods, tools and procedures , 2013, Inf. Vis..

[51]  Paul Parsons,et al.  Reexamining the cognitive utility of 3D visualizations using augmented reality holograms , 2017 .

[52]  Raimund Dachselt,et al.  GIAnT: Visualizing Group Interaction at Large Wall Displays , 2017, CHI.

[53]  Andrea Palazzi,et al.  Learning to Detect and Track Visible and Occluded Body Joints in a Virtual World , 2018, ECCV.

[54]  Niklas Elmqvist,et al.  There Is No Spoon: Evaluating Performance, Space Use, and Presence with Expert Domain Users in Immersive Analytics , 2019, IEEE Transactions on Visualization and Computer Graphics.

[55]  Aidong Lu,et al.  Improving Information Sharing and Collaborative Analysis for Remote GeoSpatial Visualization Using Mixed Reality , 2019, 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[56]  Bruce H. Thomas,et al.  ImAxes: Immersive Axes as Embodied Affordances for Interactive Multivariate Data Visualisation , 2017, UIST.

[57]  Bruce H. Thomas,et al.  GeoGate: Correlating Geo-Temporal Datasets Using an Augmented Reality Space-Time Cube and Tangible Interactions , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[58]  Christian Theobalt,et al.  Single-Shot Multi-person 3D Pose Estimation from Monocular RGB , 2017, 2018 International Conference on 3D Vision (3DV).

[59]  John T. Stasko,et al.  Toward a Deeper Understanding of the Role of Interaction in Information Visualization , 2007, IEEE Transactions on Visualization and Computer Graphics.

[60]  Deb Roy,et al.  An immersive system for browsing and visualizing surveillance video , 2010, ACM Multimedia.

[61]  M. Sheelagh T. Carpendale,et al.  VisTACO: visualizing tabletop collaboration , 2010, ITS '10.

[62]  Xiaoru Yuan,et al.  TripVista: Triple Perspective Visual Trajectory Analytics and its application on microscopic traffic data at a road intersection , 2011, 2011 IEEE Pacific Visualization Symposium.

[63]  Christian Tominski,et al.  Visualization of Trajectory Attributes in Space–Time Cube and Trajectory Wall , 2014 .

[64]  M. Sheelagh T. Carpendale,et al.  A Review of Temporal Data Visualizations Based on Space-Time Cube Operations , 2014, EuroVis.

[65]  Magy Seif El-Nasr,et al.  Visualizing and understanding players' behavior in video games: discovering patterns and supporting aggregation and comparison , 2011, SIGGRAPH 2011.

[66]  Harald Reiterer,et al.  Clusters, Trends, and Outliers: How Immersive Technologies Can Facilitate the Collaborative Analysis of Multidimensional Data , 2018, CHI.

[67]  Raimund Dachselt,et al.  Investigating Smartphone-based Pan and Zoom in 3D Data Spaces in Augmented Reality , 2019, MobileHCI.

[68]  Jie Li,et al.  Using Virtual Reality Technique to Enhance Experience of Exploring 3D Trajectory Visualizations , 2015, VINCI.