How to Safely Augment Reality: Challenges and Directions

Augmented reality (AR) technologies, such as those in head-mounted displays like Microsoft HoloLens or in automotive windshields, are poised to change how people interact with their devices and the physical world. Though researchers have begun considering the security, privacy, and safety issues raised by these technologies, to date such efforts have focused on input, i.e., how to limit the amount of private information to which AR applications receive access. In this work, we focus on the challenge of output management: how can an AR operating system allow multiple concurrently running applications to safely augment the user's view of the world? That is, how can the OS prevent apps from (for example) interfering with content displayed by other apps or the user's perception of critical real-world context, while still allowing them sufficient flexibility to implement rich, immersive AR scenarios? We explore the design space for the management of visual AR output, propose a design that balances OS control with application flexibility, and lay out the research directions raised and enabled by this proposal.

[1]  Ronald Azuma,et al.  A survey of augmented reality" Presence: Teleoperators and virtual environments , 1997 .

[2]  Jonathan S. Shapiro,et al.  Design of the EROS Trusted Window System , 2004, USENIX Security Symposium.

[3]  Helen J. Wang,et al.  Enabling Fine-Grained Permissions for Augmented Reality Applications with Recognizers , 2013, USENIX Security Symposium.

[4]  Morten Fjeld,et al.  Mixed Reality: A Survey , 2009, Human Machine Interaction.

[5]  Henry Been-Lirn Duh,et al.  Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[6]  Vitaly Shmatikov,et al.  A Scanner Darkly: Protecting User Privacy from Perceptual Applications , 2013, 2013 IEEE Symposium on Security and Privacy.

[7]  David J. Crandall,et al.  PlaceAvoider: Steering First-Person Cameras away from Sensitive Spaces , 2014, NDSS.

[8]  Ashwin Machanavajjhala,et al.  MarkIt: privacy markers for protecting visual secrets , 2014, UbiComp Adjunct.

[9]  Dieter Schmalstieg,et al.  The Studierstube Augmented Reality Project , 2002, Presence: Teleoperators & Virtual Environments.

[10]  Victor Ng-Thow-Hing,et al.  A left-turn driving aid using projected oncoming vehicle paths with augmented reality , 2013, AutomotiveUI.

[11]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[12]  Frank Dürr,et al.  An access control concept for novel automotive HMI systems , 2014, SACMAT '14.

[13]  Helen J. Wang,et al.  SurroundWeb: Mitigating Privacy Concerns in a 3D Web Browser , 2015, 2015 IEEE Symposium on Security and Privacy.

[14]  D. W. F. van Krevelen,et al.  A Survey of Augmented Reality Technologies, Applications and Limitations , 2010, Int. J. Virtual Real..

[15]  John McHugh,et al.  Evolution of a trusted B3 window system prototype , 1992, Proceedings 1992 IEEE Computer Society Symposium on Research in Security and Privacy.

[16]  Ivan E. Sutherland,et al.  A head-mounted three dimensional display , 1968, AFIPS Fall Joint Computing Conference.

[17]  Tadayoshi Kohno,et al.  Securing Embedded User Interfaces: Android and Beyond , 2013, USENIX Security Symposium.

[18]  Helen J. Wang,et al.  World-Driven Access Control for Continuous Sensing , 2014, CCS.

[19]  Tadayoshi Kohno,et al.  Security and privacy for augmented reality systems , 2014, Commun. ACM.

[20]  Steven K. Feiner,et al.  Enveloping users and computers in a collaborative 3D augmented reality , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[21]  George Papagiannakis,et al.  A survey of mobile and wireless technologies for augmented reality systems , 2008, Comput. Animat. Virtual Worlds.

[22]  Helen J. Wang,et al.  Operating System Support for Augmented Reality Applications , 2013, HotOS.

[23]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.