ILLIXR: Enabling End-to-End Extended Reality Research

An increasing number of edge systems have large computational demands, stringent resource constraints, and end-to-end quality-driven goodness metrics. Architects have embraced domain-specific accelerators to meet the demands of such systems. We make the case for research that shifts emphasis from domain-specific accelerators to domain-specific systems, with a consequent shift from evaluations using benchmarks that are collections of independent applications to those using testbeds that are full integrated systems. We describe extended reality (XR) as an exciting domain motivating such domain-specific systems research, but hampered by the lack of an end-to-end evaluation testbed. We present ILLIXR (Illinois Extended Reality testbed), the first fully open source XR system and research testbed. ILLIXR enables system innovations with end-to-end co-designed hardware, compiler, OS, and algorithm, and driven by end-user perceived quality-of-experience (QoE) metrics. Using ILLIXR, we perform the first comprehensive quantitative analysis of performance, power, and QoE for a complete XR system and its individual components. We describe several implications of our results that propel new directions in architecture, systems, and algorithm research for domain-specific systems in general, and XR in particular, all enabled by ILLIXR.

[1]  Youngki Lee,et al.  Heimdall: mobile GPU coordination platform for augmented reality applications , 2020, MobiCom.

[2]  Michael F. P. O'Boyle,et al.  SLAMBench2: Multi-Objective Head-to-Head Benchmarking for Visual SLAM , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[3]  Mehdi Bennis,et al.  Toward Low-Latency and Ultra-Reliable Virtual Reality , 2018, IEEE Network.

[4]  Jing Wang,et al.  Processing-in-Memory Enabled Graphics Processors for 3D Rendering , 2017, 2017 IEEE International Symposium on High Performance Computer Architecture (HPCA).

[5]  David Kanter.  GRAPHICS PROCESSING REQUIREMENTS FOR ENABLING IMMERSIVE VR , 2015 .

[6]  Giancarlo Ruocco,et al.  Computer generation of optimal holograms for optical trap arrays. , 2007, Optics express.

[7]  Jóakim von Kistowski,et al.  SPEC CPU2017: Next-Generation Compute Benchmark , 2018, ICPE Companion.

[8]  Xin Liu,et al.  Motion-Prediction-Based Multicast for 360-Degree Video Transmissions , 2017, 2017 14th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON).

[9]  Robert LiKamWa,et al.  Rhythmic pixel regions: multi-resolution visual sensing system towards high-precision visual computing at low power , 2021, ASPLOS.

[10]  Stefanos Kaxiras,et al.  Splash-3: A properly synchronized benchmark suite for contemporary research , 2016, 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS).

[11]  J. M. P. van Waveren,et al.  The asynchronous time warp for virtual reality on consumer hardware , 2016, VRST.

[12]  Feng Qian,et al.  Flare: Practical Viewport-Adaptive 360-Degree Video Streaming for Mobile Devices , 2018, MobiCom.

[13]  B. V. K. Vijaya Kumar,et al.  Towards multifocal displays with dense focal stacks , 2018, ACM Trans. Graph..

[14]  Parthasarathy Ranganathan,et al.  Warehouse-scale video acceleration: co-design and deployment in the wild , 2021, ASPLOS.

[15]  Samir R. Das,et al.  Streaming 360-Degree Videos Using Super-Resolution , 2020, IEEE INFOCOM 2020 - IEEE Conference on Computer Communications.

[16]  Roland Siegwart,et al.  The EuRoC micro aerial vehicle datasets , 2016, Int. J. Robotics Res..

[17]  Yong Man Ro,et al.  VR IQA NET: Deep Virtual Reality Image Quality Assessment Using Adversarial Learning , 2018, 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[18]  Matthew Mattina,et al.  Euphrates: Algorithm-SoC Co-Design for Low-Power Mobile Continuous Vision , 2018, 2018 ACM/IEEE 45th Annual International Symposium on Computer Architecture (ISCA).

[19]  Tomas Akenine-Möller,et al.  FLIP: A Difference Evaluator for Alternating Images , 2020, Proc. ACM Comput. Graph. Interact. Tech..

[20]  Yu Feng,et al.  Mesorasi: Architecture Support for Point Cloud Analytics via Delayed-Aggregation , 2020, 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[21]  Mattias Goksör,et al.  Real-time generation of fully optimized holograms for optical trapping applications , 2011, NanoScience + Engineering.

[22]  Alan C. Bovik,et al.  Learning to Predict Streaming Video QoE: Distortions, Rebuffering and Memory , 2017, ArXiv.

[23]  Jonathan T. Barron,et al.  A hardware-friendly bilateral solver for real-time virtual reality video , 2017, High Performance Graphics.

[24]  David Chu,et al.  FlashBack: Immersive Virtual Reality on Mobile Devices via Rendering Memoization , 2016, MobiSys.

[25]  Xing Liu,et al.  Firefly: Untethered Multi-user VR for Commodity Mobile Devices , 2020, USENIX ATC.

[26]  An Introduction to Higher-Order Ambisonic , 2005 .

[27]  John L. Henning.  SPEC CPU2006 benchmark descriptions , 2006, CARN.

[28]  Santhosh Kumar Rethinagiri,et al.  Visual Inertial Odometry At the Edge: A Hardware-Software Co-design Approach for Ultra-low Latency and Power , 2019, 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE).

[29]  Sibendu Paul,et al.  Coterie: Exploiting Frame Similarity to Enable High-Quality Multiplayer VR on Commodity Mobile Devices , 2020, ASPLOS.

[30]  Chengde Wan,et al.  MEgATrack , 2020, ACM Trans. Graph..

[31]  Yulyani Arifin,et al.  User Experience Metric for Augmented Reality Application: A Review , 2018 .

[32]  Gregory Hughes,et al.  OpenEDS: Open Eye Dataset , 2019, ArXiv.

[33]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[34]  Wei Wei,et al.  2019 Formatting Instructions for Authors Using LaTeX , 2018 .

[35]  Anoop Gupta,et al.  The SPLASH-2 programs: characterization and methodological considerations , 1995, ISCA.

[36]  David Black-Schaffer,et al.  A graphics tracing framework for exploring CPU+GPU memory systems , 2017, 2017 IEEE International Symposium on Workload Characterization (IISWC).

[37]  Michael Taylor,et al.  Q-VR: system-level design for future mobile collaborative virtual reality , 2021, ASPLOS.

[38]  Kevin Skadron,et al.  Rodinia: A benchmark suite for heterogeneous computing , 2009, 2009 IEEE International Symposium on Workload Characterization (IISWC).

[39]  Wei Gao,et al.  MUVR: Supporting Multi-User Mobile Virtual Reality with Resource Constrained Edge Cloud , 2018, 2018 IEEE/ACM Symposium on Edge Computing (SEC).

[40]  Patrick Seeling,et al.  Towards Predictions of the Image Quality of Experience for Augmented Reality Scenarios , 2017, ArXiv.

[41]  Andrew Hines,et al.  AMBIQUAL: Towards a Quality Metric for Headphone Rendered Compressed Ambisonic Spatial Audio , 2020 .

[42]  Michael Gaebler,et al.  Multidimensional Evaluation of Virtual Reality Paradigms in Clinical Neuropsychology: Application of the VR-Check Framework , 2020, Journal of medical Internet research.

[43]  Rajesh Aggarwal,et al.  An Evidence-Based Virtual Reality Training Program for Novice Laparoscopic Surgeons , 2006, Annals of surgery.

[44]  Wei Wang,et al.  A Benchmarking Framework for Interactive 3D Applications in the Cloud , 2020, 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO).

[45]  Jian Huang,et al.  Semantic-Aware Virtual Reality Video Streaming , 2018, APSys.

[46]  Bernd Fröhlich,et al.  Efficient Hybrid Image Warping for High Frame-Rate Stereoscopic Rendering , 2017, IEEE Transactions on Visualization and Computer Graphics.

[47]  Reynold J. Bailey,et al.  RITnet: Real-time Semantic Segmentation of the Eye for Gaze Tracking , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[48]  Tao Li,et al.  Understanding the Characteristics of Mobile Augmented Reality Applications , 2018, 2018 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS).

[49]  Michael F. P. O'Boyle,et al.  SLAMBench 3.0: Systematic Automated Reproducible Evaluation of SLAM Systems for Robot Vision Challenges and Scene Understanding , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[50]  Sujit Dey,et al.  Predictive View Generation to Enable Mobile 360-degree and VR Experiences , 2018, VR/AR Network@SIGCOMM.

[51]  Daniel Flores Quiros,et al.  Project Esky: Enabling High Fidelity Augmented Reality on an Open Source Platform , 2020, ISS Companion.

[52]  Yong Man Ro,et al.  Deep Virtual Reality Image Quality Assessment With Human Perception Guider for Omnidirectional Image , 2020, IEEE Transactions on Circuits and Systems for Video Technology.

[53]  Qiang Liu,et al.  Eudoxus: Characterizing and Accelerating Localization in Autonomous Machines Industry Track Paper , 2020, 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA).

[54]  Yuhao Zhu,et al.  Tigris: Architecture and Algorithms for 3D Perception in Point Clouds , 2019, MICRO.

[55]  Luca Carlone,et al.  Navion: A 2-mW Fully Integrated Real-Time Visual-Inertial Odometry Accelerator for Autonomous Navigation of Nano Drones , 2018, IEEE Journal of Solid-State Circuits.

[56]  Amrita Mazumdar,et al.  Exploring computation-communication tradeoffs in camera systems , 2017, 2017 IEEE International Symposium on Workload Characterization (IISWC).

[57]  Alois Sontacchi,et al.  Producing 3D Audio in Ambisonics , 2015 .

[58]  Tor M. Aamodt,et al.  Analyzing Machine Learning Workloads Using a Detailed GPU Simulator , 2018, 2019 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS).

[59]  Shu Yang,et al.  Subjective and objective quality assessment of panoramic videos in virtual reality environments , 2017, 2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[60]  Y. Charlie Hu,et al.  Furion: Engineering High-Quality Immersive Virtual Reality on Today's Mobile Devices , 2017, IEEE Transactions on Mobile Computing.

[61]  Kai Li,et al.  The PARSEC benchmark suite: Characterization and architectural implications , 2008, 2008 International Conference on Parallel Architectures and Compilation Techniques (PACT).

[62]  Michael F. P. O'Boyle,et al.  Introducing SLAMBench, a performance and accuracy benchmarking methodology for SLAM , 2014, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[63]  M. Zaharia,et al.  Future Directions for Parallel and Distributed Computing: SPX 2019 Workshop Report , 2019 .

[64]  Shuaiwen Song,et al.  PIM-VR: Erasing Motion Anomalies In Highly-Interactive Virtual Reality World with Customized Memory Cube , 2019, 2019 IEEE International Symposium on High Performance Computer Architecture (HPCA).

[65]  Yuhao Zhu,et al.  Energy-Efficient Video Processing for Virtual Reality , 2019, 2019 ACM/IEEE 46th Annual International Symposium on Computer Architecture (ISCA).

[66]  Bin Jiang,et al.  3D Panoramic Virtual Reality Video Quality Assessment Based on 3D Convolutional Neural Networks , 2018, IEEE Access.

[67]  Ryan Kastner,et al.  FPGA Architectures for Real-time Dense SLAM , 2019, 2019 IEEE 30th International Conference on Application-specific Systems, Architectures and Processors (ASAP).

[68]  Yuhao Zhu,et al.  ASV: Accelerated Stereo Vision System , 2019, MICRO.

[69]  Anoop Gupta,et al.  SPLASH: Stanford parallel applications for shared-memory , 1992, CARN.

[70]  Mahmut T. Kandemir,et al.  Déjà View: Spatio-Temporal Compute Reuse for‘ Energy-Efficient 360° VR Video Streaming , 2020, 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA).

[71]  Rob A. Rutenbar,et al.  Error resilient MRF message passing architecture for stereo matching , 2013, SiPS 2013 Proceedings.