Ensuring Safety in Augmented Reality from Trade-off Between Immersion and Situation Awareness

Although the mobility and emerging technology of augmented reality (AR) have brought significant entertainment and convenience in everyday life, the use of AR is becoming a social problem as the accidents caused by a shortage of situation awareness due to an immersion of AR are increasing. In this paper, we address the trade-off between immersion and situation awareness as the fundamental factor of the AR-related accidents. As a solution against the trade-off, we propose a third-party component that prevents pedestrian-vehicle accidents in a traffic environment based on vehicle position estimation (VPE) and vehicle position visualization (VPV). From a RGB image sequence, VPE efficiently estimates the relative 3D position between a user and a car using generated convolutional neural network (CNN) model with a region-of-interest based scheme. VPV shows the estimated car position as a dot using an out-of-view object visualization method to alert the user from possible collisions. The VPE experiment with 16 combinations of parameters showed that the InceptionV3 model, fine-tuned on activated images yields the best performance with a root mean squared error of 0.34 m in 2.1 ms. The user study of VPV showed the inversely proportional relationship between the immersion controlled by the difficulty of the AR game and the frequency of situation awareness in both quantitatively and qualitatively. Additional VPV experiment assessing two out-of-view object visualization methods (EyeSee360 and Radar) showed no significant effect on the participants' activity, while EyeSee360 yielded faster responses and Radar engendered participants' preference on average. Our field study demonstrated an integration of VPE and VPV which has potentials for safety-ensured immersion when the proposed component is used for AR in daily uses. We expect that when the proposed component is developed enough to be used in real world, it will contribute to the safety-ensured AR, as well as to the population of AR.

[1]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Marco Gruteser,et al.  On the limits of positioning-based pedestrian risk awareness , 2014, MARS '14.

[3]  S. Thorpe,et al.  Speed of processing in the human visual system , 1996, Nature.

[4]  Bernard Ghanem,et al.  UE4Sim: A Photo-Realistic Simulator for Computer Vision Applications , 2017, ArXiv.

[5]  Bellal Joseph,et al.  Potential perils of peri-Pokémon perambulation: the dark reality of augmented reality? , 2016, Oxford medical case reports.

[6]  Richard Wener,et al.  Mobile telephones, distracted attention, and pedestrian safety. , 2008, Accident; analysis and prevention.

[7]  Nassir Navab,et al.  Deeper Depth Prediction with Fully Convolutional Residual Networks , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[8]  Rob Fergus,et al.  Depth Map Prediction from a Single Image using a Multi-Scale Deep Network , 2014, NIPS.

[9]  Fumio Kishino,et al.  Augmented reality: a class of displays on the reality-virtuality continuum , 1995, Other Conferences.

[10]  Ming-Yu Liu,et al.  Deep 360 Pilot: Learning a Deep Agent for Piloting through 360° Sports Videos , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Rossella Snenghi,et al.  Pedestrian Inattention Blindness While Playing Pokémon Go as an Emerging Health-Risk Behavior: A Case Report , 2017, Journal of medical Internet research.

[12]  Dalibor Pešić,et al.  The effects of mobile phone use on pedestrian crossing behaviour at unsignalized intersections – Models for predicting unsafe pedestrians behaviour , 2016 .

[13]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[14]  Leopoldo Armesto,et al.  Haptic Feedback to Assist Bus Drivers for Pedestrian Safety at Low Speed , 2016, IEEE Transactions on Haptics.

[15]  Heni Ben Amor,et al.  Deep Predictive Models for Collision Risk Assessment in Autonomous Driving , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[16]  Moises Sudit,et al.  ViSAw: Visualizing threat and impact assessment for enhanced situation awareness , 2009, MILCOM 2009 - 2009 IEEE Military Communications Conference.

[17]  Mark Dredze,et al.  Pokémon GO-A New Distraction for Drivers and Pedestrians. , 2016, JAMA internal medicine.

[18]  Marco Gruteser,et al.  LookUp: Enabling Pedestrian Safety Services via Shoe Sensing , 2015, MobiSys.

[19]  Doug A. Bowman,et al.  Virtual Reality: How Much Immersion Is Enough? , 2007, Computer.

[20]  Tomoyuki Ishida,et al.  Mobile Traffic Accident Prevention System based on Chronological Changes of Wireless Signals and Sensors , 2017, J. Wirel. Mob. Networks Ubiquitous Comput. Dependable Appl..

[21]  Moritz Kampelmühler,et al.  Camera-based vehicle velocity estimation from monocular video , 2018, ArXiv.

[22]  Guosheng Lin,et al.  Deep convolutional neural fields for depth estimation from a single image , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Antonio M. López,et al.  The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Aurelio Ponz,et al.  IVVI 2.0: An intelligent vehicle based on computational perception , 2014, Expert Syst. Appl..

[25]  Hoon Kim,et al.  Crash To Not Crash: Playing Video Games To Predict Vehicle Collisions , 2017, ICML 2017.

[26]  Sun-Young Hwang,et al.  Robust Range Estimation with a Monocular Camera for Vision-Based Forward Collision Warning System , 2014, TheScientificWorldJournal.

[27]  Susanne Boll,et al.  Visualizing out-of-view objects in head-mounted augmented reality , 2017, MobileHCI.

[28]  Oliver Deussen,et al.  Traffic visualization in helmet-mounted displays in synchronization with navigation displays , 2015, 2015 IEEE/AIAA 34th Digital Avionics Systems Conference (DASC).

[29]  A. Ali,et al.  Distance estimation and vehicle position detection based on monocular camera , 2016, 2016 Al-Sadeq International Conference on Multidisciplinary in IT and Communication Science and Applications (AIC-MITCSA).

[30]  Pourang Irani,et al.  Comparing visualizations for tracking off-screen moving targets , 2007, CHI Extended Abstracts.

[31]  Christer Hydén,et al.  Estimating the severity of safety related behaviour. , 2006, Accident; analysis and prevention.

[32]  Susan T Chrysler,et al.  Creating Pedestrian Crash Scenarios in a Driving Simulator Environment , 2015, Traffic injury prevention.

[33]  Helbing,et al.  Congested traffic states in empirical observations and microscopic simulations , 2000, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[34]  Hong Z. Tan,et al.  To Go or Not to Go: Stimulus-Response Compatibility for Tactile and Auditory Pedestrian Collision Warnings , 2009, IEEE Transactions on Haptics.

[35]  Despina Stavrinos,et al.  Distracted walking: cell phones increase injury risk for college pedestrians. , 2011, Journal of safety research.

[36]  Douglas C. Schmidt,et al.  WreckWatch: Automatic Traffic Accident Detection and Notification with Smartphones , 2011, Mob. Networks Appl..

[37]  Thierry Chateau,et al.  Deep MANTA: A Coarse-to-Fine Many-Task Network for Joint 2D and 3D Vehicle Analysis from Monocular Image , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Bernard Ghanem,et al.  Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications , 2017, International Journal of Computer Vision.

[39]  Andreas Geiger,et al.  Vision meets robotics: The KITTI dataset , 2013, Int. J. Robotics Res..

[40]  Susanne Boll,et al.  EyeSee360: designing a visualization technique for out-of-view objects in head-mounted augmented reality , 2017, SUI.

[41]  Johannes Schöning,et al.  The Geography of Pokémon GO: Beneficial and Problematic Effects on Places and Movement , 2017, CHI.

[42]  Vladlen Koltun,et al.  Playing for Data: Ground Truth from Computer Games , 2016, ECCV.

[43]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[44]  Antonio Corradi,et al.  WalkSafe: a pedestrian safety app for mobile phone users who walk and talk while crossing roads , 2012, HotMobile '12.