Aristo: An Augmented Reality Platform for Immersion and Interactivity

This paper introduces our augmented reality platform, Aristo, which aims to provide users with physical feedback when interacting with virtual objects. We use Vivepaper, a product we launched on Aristo in 2016, to illustrate the platform's performance requirements and key algorithms. We specifically depict Vivepaper's tracking and gesture recognition algorithms, which involve several trade-offs between speed and accuracy to achieve an immersive experience.

[1]  Vincent Lepetit,et al.  Scalable real-time planar targets tracking for digilog books , 2010, The Visual Computer.

[2]  Xiaolong Zhu,et al.  Pixel-Level Hand Detection with Shape-Aware Structured Forests , 2014, ACCV.

[3]  Bingbing Ni,et al.  Cascaded Interactional Targeting Network for Egocentric Video Analysis , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Ying Wu,et al.  Visual panel: virtual mouse, keyboard and 3D controller with an ordinary piece of paper , 2001, PUI '01.

[5]  Francisco José Madrid-Cuevas,et al.  Automatic generation and detection of highly reliable fiducial markers under occlusion , 2014, Pattern Recognit..

[6]  Ulrich Schwanecke,et al.  Real-Time Monocular Segmentation and Pose Tracking of Multiple Objects , 2016, ECCV.

[7]  Dieter Schmalstieg,et al.  Making Augmented Reality Practical on Mobile Phones, Part 1 , 2009, IEEE Computer Graphics and Applications.

[8]  Junsong Yuan,et al.  Barehanded music: real-time hand interaction for virtual piano , 2016, I3D.

[9]  Nikolaos G. Bourbakis,et al.  A survey of skin-color modeling and detection methods , 2007, Pattern Recognit..

[10]  Shogo Nishida,et al.  Virtual Pop-Up Book Based on Augmented Reality , 2007, HCI.

[11]  D. Williamson,et al.  The box plot: a simple visual method to interpret data. , 1989, Annals of internal medicine.

[12]  Woontack Woo,et al.  Digilog book for temple bell tolling experience based on interactive augmented reality , 2011, Virtual Reality.

[13]  Woontack Woo,et al.  Enhancing Immersiveness in AR-Based Product Design , 2006, ICAT.

[14]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[15]  Kris M. Kitani,et al.  Going Deeper into First-Person Activity Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  R. Bencina,et al.  Improved Topological Fiducial Tracking in the reacTIVision System , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops.

[17]  Victor S. Lempitsky,et al.  Learnable Visual Markers , 2016, NIPS.

[18]  Karthik Ramani,et al.  DeepHand: Robust Hand Pose Estimation by Completing a Matrix Imputed with Deep Features , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  Éric Marchand,et al.  Real-time markerless tracking for augmented reality: the virtual visual servoing framework , 2006, IEEE Transactions on Visualization and Computer Graphics.

[20]  Vincent Lepetit,et al.  The haunted book , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[21]  Cheng Li,et al.  Pixel-Level Hand Detection in Ego-centric Videos , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[22]  Keiichi Abe,et al.  Topological structural analysis of digitized binary images by border following , 1985, Comput. Vis. Graph. Image Process..

[23]  Shuchang Xu,et al.  Wrist Recognition and the Center of the Palm Estimation Based on Depth Camera , 2013, 2013 International Conference on Virtual Reality and Visualization.

[24]  Poonsri Vate-U-Lan,et al.  An Augmented Reality 3D Pop-Up Book: The Development of a Multimedia Project for English Language Teaching , 2012, 2012 IEEE International Conference on Multimedia and Expo.

[25]  Edward Y. Chang,et al.  CLKN: Cascaded Lucas-Kanade Networks for Image Alignment , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Edwin Olson,et al.  AprilTag: A robust and flexible visual fiducial system , 2011, 2011 IEEE International Conference on Robotics and Automation.

[27]  Mark Fiala,et al.  ARTag, a fiducial marker system using digital techniques , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[28]  G. Roth,et al.  Hand tracking for interactive pattern-based augmented reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[29]  Matthias Rauterberg,et al.  Left/Right Hand Segmentation in Egocentric Videos , 2016, 1607.06264.

[30]  Edwin Olson,et al.  AprilTag 2: Efficient and robust fiducial detection , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[31]  Éric Marchand,et al.  Pose Estimation for Augmented Reality: A Hands-On Survey , 2016, IEEE Transactions on Visualization and Computer Graphics.

[32]  Luca Benini,et al.  Gesture Recognition in Ego-centric Videos Using Dense Trajectories and Hand Segmentation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[33]  Anupam Agrawal,et al.  Vision based hand gesture recognition for human computer interaction: a survey , 2012, Artificial Intelligence Review.

[34]  Matthias Rauterberg,et al.  GPU Accelerated Left/Right Hand-Segmentation in First Person Vision , 2016, ECCV Workshops.

[35]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[36]  Mark Fiala,et al.  Comparing ARTag and ARToolkit Plus fiducial marker systems , 2005, IEEE International Workshop on Haptic Audio Visual Environments and their Applications.

[37]  Andrea Torsello,et al.  An Accurate and Robust Artificial Marker Based on Cyclic Codes , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[38]  Hyun Seung Yang.  Realistic e-learning system based on mixed reality , 2009, VRCAI '09.

[39]  Vincent Lepetit,et al.  Fine Hand Segmentation using Convolutional Neural Networks , 2016, ArXiv.

[40]  Masa Inakage,et al.  Little red: storytelling in mixed reality , 2003, SIGGRAPH '03.

[41]  Enrico Costanza,et al.  A Region Adjacency Tree Approach to the Detection and Design of Fiducials , 2003, VVG.

[42]  Constantine Stephanidis,et al.  Augmented interaction with physical books in an Ambient Intelligence learning environment , 2013, Multimedia Tools and Applications.

[43]  Dieter Schmalstieg,et al.  ARToolKitPlus for Pose Trackin on Mobile Devices , 2007 .

[44]  Truong Q. Nguyen,et al.  Hand segmentation for hand-object interaction from depth map , 2017, 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP).