Towards Pervasive Augmented Reality: Context-Awareness in Augmented Reality

Augmented Reality is a technique that enables users to interact with their physical environment through the overlay of digital information. While being researched for decades, more recently, Augmented Reality moved out of the research labs and into the field. While most of the applications are used sporadically and for one particular task only, current and future scenarios will provide a continuous and multi-purpose user experience. Therefore, in this paper, we present the concept of Pervasive Augmented Reality, aiming to provide such an experience by sensing the user’s current context and adapting the AR system based on the changing requirements and constraints. We present a taxonomy for Pervasive Augmented Reality and context-aware Augmented Reality, which classifies context sources and context targets relevant for implementing such a context-aware, continuous Augmented Reality experience. We further summarize existing approaches that contribute towards Pervasive Augmented Reality. Based our taxonomy and survey, we identify challenges for future research directions in Pervasive Augmented Reality.

[1]  E. Hall,et al.  The Hidden Dimension , 1970 .

[2]  Ivan E. Sutherland,et al.  A head-mounted three dimensional display , 1968, AFIPS Fall Joint Computing Conference.

[3]  R. Weale Vision. A Computational Investigation Into the Human Representation and Processing of Visual Information. David Marr , 1983 .

[4]  Donald A. Norman,et al.  Stages and Levels in Human-Machine Interaction , 1984, Int. J. Man Mach. Stud..

[5]  John Scott What is social network analysis , 2010 .

[6]  Gregory D. Abowd,et al.  Formal Aspects of Human-computer Interaction , 1991 .

[7]  T. P. Caudell,et al.  Augmented reality: an application of heads-up display technology to manual manufacturing processes , 1992, Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences.

[8]  Tom Carey,et al.  ACM SIGCHI Curricula for Human-Computer Interaction , 1992 .

[9]  A. Strauss,et al.  Basics of qualitative research: Grounded theory procedures and techniques. , 1992 .

[10]  George W. Fitzmaurice,et al.  Situated information spaces and spatially aware palmtop computers , 1993, CACM.

[11]  Pamela Jordan Basics of qualitative research: Grounded theory procedures and techniques , 1994 .

[12]  Bill Hillier,et al.  Space is the machine: A configurational theory of architecture , 1996 .

[13]  Jennifer Healey,et al.  Augmented Reality through Wearable Computing , 1997, Presence: Teleoperators & Virtual Environments.

[14]  Jun Rekimoto,et al.  NaviCam:A Magnifying Glass Approach to Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[15]  J. Henderson,et al.  High-level scene perception. , 1999, Annual review of psychology.

[16]  Albrecht Schmidt,et al.  There is more to context than location , 1999, Comput. Graph..

[17]  Gregory D. Abowd,et al.  Towards a Better Understanding of Context and Context-Awareness , 1999, HUC.

[18]  Joëlle Coutaz,et al.  Plasticity of User Interfaces: Framework and Research Agenda , 1999, INTERACT.

[19]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[20]  Steven K. Feiner,et al.  Information filtering for mobile augmented reality , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[21]  Jonathan Trevor,et al.  Exploiting space and location as a design framework for interactive mobile systems , 2000, TCHI.

[22]  Nassir Navab,et al.  Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[23]  Guanling Chen,et al.  A Survey of Context-Aware Mobile Computing Research , 2000 .

[24]  Ivan Poupyrev,et al.  Virtual object manipulation on a table-top AR environment , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[25]  Steven K. Feiner,et al.  View management for virtual and augmented reality , 2001, UIST '01.

[26]  Kirk Martinez,et al.  Adaptive Hypermedia in Augmented Reality , 2001 .

[27]  Paul Dourish,et al.  Seeking a Foundation for Context-Aware Computing , 2001, Hum. Comput. Interact..

[28]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[29]  Jannick P. Rolland,et al.  A Survey of Tracking Technologies for Virtual Environments , 2001 .

[30]  Hiroyuki Ohno,et al.  An optical see-through display for mutual occlusion with a real-time stereovision system , 2001, Comput. Graph..

[31]  Greg Welch,et al.  Shader Lamps: Animating Real Objects With Image-Based Illumination , 2001, Rendering Techniques.

[32]  Dag Svanæs,et al.  Context-Aware Technology: A Phenomenological Perspective , 2001, Hum. Comput. Interact..

[33]  Saul Greenberg,et al.  Context as a Dynamic Construct , 2001, Hum. Comput. Interact..

[34]  Greg Welch,et al.  Motion Tracking: No Silver Bullet, but a Respectable Arsenal , 2002, IEEE Computer Graphics and Applications.

[35]  Steven K. Feiner,et al.  An annotated situation-awareness aid for augmented reality , 2002, UIST '02.

[36]  Blair MacIntyre,et al.  Estimating and adapting to registration errors in augmented reality systems , 2002, Proceedings IEEE Virtual Reality 2002.

[37]  Antti Oulasvirta,et al.  Understanding Mobile Contexts , 2003, Mobile HCI.

[38]  David E. Millard,et al.  Augmented reality as an interface to adaptive hypermedia systems , 2003, New Rev. Hypermedia Multim..

[39]  Paul Dourish,et al.  What we talk about when we talk about context , 2004, Personal and Ubiquitous Computing.

[40]  Jadwiga Indulska,et al.  A software engineering framework for context-aware pervasive computing , 2004, Second IEEE Annual Conference on Pervasive Computing and Communications, 2004. Proceedings of the.

[41]  Claudia Linnhoff-Popien,et al.  A Context Modeling Survey , 2004 .

[42]  Stephen DiVerdi,et al.  Level of detail interfaces , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[43]  Fan Xiao,et al.  Display-relative calibration for optical see-through head-mounted displays , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[44]  Dieter Schmalstieg,et al.  Agents that talk and hit back: animated agents in augmented reality , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[45]  Ivan Poupyrev,et al.  3D User Interfaces: Theory and Practice , 2004 .

[46]  Steven K. Feiner,et al.  BRIDGING THE GAPS: HYBRID TRACKING FOR ADAPTIVE MOBILE AUGMENTED REALITY , 2004, Appl. Artif. Intell..

[47]  Blair MacIntyre,et al.  OSGAR: a scene graph with uncertain transformations , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[48]  J. Edward Swan,et al.  Survey of User-Based Experimentation in Augmented Reality , 2005 .

[49]  Asa MacWilliams,et al.  A decentralized adaptive architecture for ubiquitous augmented reality systems , 2005 .

[50]  Holger Regenbrecht,et al.  Augmented reality projects in the automotive and aerospace industries , 2005, IEEE Computer Graphics and Applications.

[51]  Virpi Roto,et al.  Interaction in 4-second bursts: the fragmented nature of attentional resources in mobile HCI , 2005, CHI.

[52]  Vincent Lepetit,et al.  Monocular Model-Based 3D Tracking of Rigid Objects: A Survey , 2005, Found. Trends Comput. Graph. Vis..

[53]  Mark Billinghurst,et al.  Designing augmented reality interfaces , 2005, COMG.

[54]  Tom Drummond,et al.  Real-Time Video Annotations for Augmented Reality , 2005, ISVC.

[55]  Kiyoshi Kiyokawa,et al.  A study of depth visualization techniques for virtual annotations in augmented reality , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[56]  Deborah Hix,et al.  An empirical user-based study of text drawing styles and outdoor background textures for augmented reality , 2005, IEEE Proceedings. VR 2005. Virtual Reality, 2005..

[57]  Jayfus T. Doswell,et al.  Augmented Learning: Context-Aware Mobile Augmented Reality Architecture for Learning , 2006, Sixth IEEE International Conference on Advanced Learning Technologies (ICALT'06).

[58]  Tobias Höllerer,et al.  Multimodal interaction with a wearable augmented reality system , 2006, IEEE Computer Graphics and Applications.

[59]  Andreas Zimmermann,et al.  An Operational Definition of Context , 2007, CONTEXT.

[60]  Dieter Schmalstieg,et al.  Experiences with Handheld Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[61]  Steven K. Feiner,et al.  Redefining the User Interface : Augmented Reality , 2007 .

[62]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[63]  Woontack Woo,et al.  Linking Context Modelling and Contextual Reasoning , 2007 .

[64]  Youngmin Park,et al.  Context-Aware Mobile AR System for Personalization, Selective Sharing, and Interaction of Contents in Ubiquitous Computing Environments , 2007, HCI.

[65]  Gudrun Klinker,et al.  A System Architecture for Ubiquitous Tracking Environments , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[66]  Stanley Wasserman,et al.  Social Network Analysis: Methods and Applications , 1994, Structural analysis in the social sciences.

[67]  Andrew U. Frank,et al.  Different Types of „Times“ in GIS , 2007 .

[68]  Matthias Baldauf,et al.  A survey on context-aware systems , 2007, Int. J. Ad Hoc Ubiquitous Comput..

[69]  Dieter Schmalstieg,et al.  Adaptive Augmented Reality Using Context Markup and Style Maps , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[70]  George Papagiannakis,et al.  A survey of mobile and wireless technologies for augmented reality systems , 2008, Comput. Animat. Virtual Worlds.

[71]  Dieter Schmalstieg,et al.  Pose tracking from natural features on mobile phones , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[72]  Steven K. Feiner,et al.  Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality , 2008, VRST '08.

[73]  Woontack Woo,et al.  Exploiting Context-Awareness in Augmented Reality Applications , 2008, 2008 International Symposium on Ubiquitous Virtual Reality.

[74]  Tsutomu Terada,et al.  An information layout method for an optical see-through head mounted display focusing on the viewability , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[75]  Henry Been-Lirn Duh,et al.  Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR , 2008, 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality.

[76]  Gediminas Adomavicius,et al.  Context-aware recommender systems , 2008, RecSys '08.

[77]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[78]  Woontack Woo,et al.  CAMAR 2.0: Future Direction of Context-Aware Mobile Augmented Reality , 2009, 2009 International Symposium on Ubiquitous Virtual Reality.

[79]  S. Süsstrunk,et al.  Frequency-tuned salient region detection , 2009, CVPR 2009.

[80]  Pattie Maes,et al.  SixthSense: a wearable gestural interface , 2009, SIGGRAPH ASIA Art Gallery & Emerging Technologies.

[81]  Johannes Schöning,et al.  Map torchlight: a mobile augmented reality camera projector unit , 2009, CHI Extended Abstracts.

[82]  Dieter Schmalstieg,et al.  Adaptive Visualization in Outdoor AR Displays , 2009 .

[83]  Irma Lindt,et al.  Adaptive 3D-User-Interfaces , 2009 .

[84]  S. Korea,et al.  CAMAR: Context-aware Mobile Augmented Reality in Smart Space , 2009 .

[85]  Dieter Schmalstieg,et al.  Indoor Positioning and Navigation with Camera Phones , 2009, IEEE Pervasive Computing.

[86]  Marco Marengo,et al.  Context Awareness Aims at Novel Fruition Models: Augmented Reality May be the Killer Application for Context Awareness , 2010, 2010 IEEE/IFIP International Conference on Embedded and Ubiquitous Computing.

[87]  Jadwiga Indulska,et al.  A survey of context modelling and reasoning techniques , 2010, Pervasive Mob. Comput..

[88]  Jae Yeol Lee,et al.  Hand gesture-based tangible interactions for manipulating virtual objects in a mixed reality environment , 2010 .

[89]  Jesús Fontecha,et al.  Awareness marks: adaptive services through user interactions with augmented objects , 2011, Personal and Ubiquitous Computing.

[90]  D. W. F. van Krevelen,et al.  A Survey of Augmented Reality Technologies, Applications and Limitations , 2010, Int. J. Virtual Real..

[91]  Jens Grubert,et al.  Comparative User Study of two See-through Calibration Methods , 2010, VR.

[92]  Stefanie Zollmann,et al.  Image-based ghostings for single layer occlusions in augmented reality , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[93]  Arindam Dey,et al.  An Augmented Reality X-Ray system based on visual saliency , 2010, 2010 IEEE International Symposium on Mixed and Augmented Reality.

[94]  Cynthia K. Pickering,et al.  A context-aware framework supporting complex ubiquitous scenarios with Augmented Reality enabled , 2010, 5th International Conference on Pervasive Computing and Applications.

[95]  Michela C. Tacca Commonalities between Perception and Cognition , 2011, Front. Psychology.

[96]  Holger Regenbrecht,et al.  Out of reach? — A novel AR interface approach for motor rehabilitation , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[97]  Takuji Narumi,et al.  Meta Cookie+: An Illusion-Based Gustatory Display , 2011, HCI.

[98]  Toby P. Breckon,et al.  Adaptive object placement for augmented reality use in driver assistance systems , 2011 .

[99]  Kuo-Ming Chao,et al.  A Portable Framework Design to Support User Context Aware Augmented Reality Applications , 2011, 2011 Third International Conference on Games and Virtual Worlds for Serious Applications.

[100]  Filip De Turck,et al.  Dynamic deployment and quality adaptation for mobile augmented reality applications , 2011, J. Syst. Softw..

[101]  Thomas Olsson,et al.  Online user survey on current mobile augmented reality applications , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[102]  Andreas Dünser,et al.  Towards Immersive and Adaptive Augmented Reality Exposure Treatment , 2011, Annual Review of Cybertherapy and Telemedicine.

[103]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[104]  Johannes Schöning,et al.  Falling asleep with Angry Birds, Facebook and Kindle: a large scale study on mobile application usage , 2011, Mobile HCI.

[105]  Jens Grubert,et al.  Exploring the design of hybrid interfaces for augmented posters in public spaces , 2012, NordiCHI.

[106]  Erik Duval,et al.  Context-Aware Recommender Systems for Learning: A Survey and Future Challenges , 2012, IEEE Transactions on Learning Technologies.

[107]  David G. Lowe,et al.  Perceptual Organization and Visual Recognition , 2012 .

[108]  Nenad Stojanovic,et al.  An approach for using complex event processing for adaptive augmented reality in cultural heritage domain: experience report , 2012, DEBS.

[109]  Dieter Schmalstieg,et al.  Indoor navigation with mixed reality world-in-miniature views and sparse localization on mobile devices , 2012, AVI.

[110]  Dieter Schmalstieg,et al.  Image-driven view management for augmented reality browsers , 2012, 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[111]  Didier Stricker,et al.  From Interactive to Adaptive Augmented Reality , 2012, 2012 International Symposium on Ubiquitous Virtual Reality.

[112]  David Kim,et al.  HoloDesk: direct 3d interactions with a situated see-through display , 2012, CHI.

[113]  Jens Grubert,et al.  Augmented Reality Browser Survey , 2012 .

[114]  Ann Morrison,et al.  Playing it real: magic lens and static peephole interfaces for games in a public space , 2012, Mobile HCI.

[115]  Dieter Schmalstieg,et al.  Adaptive ghosted views for Augmented Reality , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[116]  Woontack Woo,et al.  A usability study of multimodal input in an augmented reality environment , 2013, Virtual Reality.

[117]  Dieter Schmalstieg,et al.  Playing it real again: a repeated evaluation of magic lens and static peephole interfaces in public space , 2013, MobileHCI '13.

[118]  Gudrun Klinker,et al.  User awareness of tracking uncertainties in AR navigation scenarios , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[119]  Hannes Kaufmann,et al.  Differential Irradiance Caching for fast high-quality light transport between virtual and real worlds , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[120]  Jens Grubert,et al.  Augmented reality browsers: essential products or only gadgets? , 2013, CACM.

[121]  Paul Marshall,et al.  Exploring the effects of space and place on engagement with an interactive installation , 2013, CHI.

[122]  Eyal Ofek,et al.  IllumiRoom: peripheral projected illusions for interactive experiences , 2013, SIGGRAPH '13.

[123]  Horst Bischof,et al.  Augmented Reality for Construction Site Monitoring and Documentation , 2014, Proceedings of the IEEE.

[124]  Blair MacIntyre,et al.  RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units , 2014, UIST.

[125]  Blair MacIntyre,et al.  Ethereal: a toolkit for spatially adaptive augmented reality content , 2014, UIST.

[126]  Daniel J. Wigdor,et al.  Panelrama: enabling easy specification of cross-device web applications , 2014, CHI.

[127]  Rania A. Hodhod,et al.  Adaptive Augmented Reality Serious Game to Foster Problem Solving Skills , 2014, Intelligent Environments.

[128]  Dieter Schmalstieg,et al.  Next-Generation Augmented Reality Browsers: Rich, Seamless, and Adaptive , 2014, Proceedings of the IEEE.

[129]  Thierry Duval,et al.  A survey of plasticity in 3D user interfaces , 2014, 2014 IEEE 7th Workshop on Software Engineering and Architectures for Realtime Interactive Systems (SEARIS).

[130]  Daniel Cremers,et al.  Semi-dense visual odometry for AR on a smartphone , 2014, 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[131]  Jean-Pierre Jessel,et al.  Adaptive augmented reality: plasticity of augmentations , 2014, VRIC.

[132]  Dieter Schmalstieg,et al.  Efficient and robust radiance transfer for probeless photorealistic augmented reality , 2014, 2014 IEEE Virtual Reality (VR).

[133]  Dieter Schmalstieg,et al.  Hedgehog labeling: View management techniques for external labels in 3D space , 2014, 2014 IEEE Virtual Reality (VR).

[134]  Gudrun Klinker,et al.  Interaction-free calibration for optical see-through head-mounted displays based on 3D Eye localization , 2014, 2014 IEEE Symposium on 3D User Interfaces (3DUI).

[135]  Gudrun Klinker,et al.  Corneal-Imaging Calibration for Optical See-Through Head-Mounted Displays , 2015, IEEE Transactions on Visualization and Computer Graphics.

[136]  Mark Billinghurst,et al.  A Survey of Augmented Reality , 2015, Found. Trends Hum. Comput. Interact..

[137]  Tobias Langlotz,et al.  The History of Mobile Augmented Reality , 2015, ArXiv.

[138]  Dieter Schmalstieg,et al.  The utility of Magic Lens interfaces on handheld devices for touristic map navigation , 2015, Pervasive Mob. Comput..

[139]  Christian Sandor,et al.  Subjective Evaluation of a Semi-Automatic Optical See-Through Head-Mounted Display Calibration Technique , 2015, IEEE Transactions on Visualization and Computer Graphics.

[140]  Jens Grubert,et al.  Perceptual and Social Challenges in Body Proximate Display Ecosystems , 2015, MobileHCI Adjunct.

[141]  Xiaolin Li,et al.  Enabling Context-Aware Indoor Augmented Reality via Smartphone Sensing and Vision Tracking , 2015, ACM Trans. Multim. Comput. Commun. Appl..

[142]  Eyal Ofek,et al.  FoveAR: Combining an Optically See-Through Near-Eye Display with Projector-Based Spatial Augmented Reality , 2015, UIST.

[143]  Jens Grubert,et al.  Design and Technology Challenges for Body Proximate Display Ecosystems , 2015, MobileHCI Adjunct.

[144]  Dieter Schmalstieg,et al.  MultiFi: Multi Fidelity Interaction with Displays On and Around the Body , 2015, CHI.

[145]  Eric E. Sabelman,et al.  The real-life dangers of augmented reality , 2015, IEEE Spectrum.

[146]  Nicolai Marquardt,et al.  WatchConnect: A Toolkit for Prototyping Smartwatch-Centric Cross-Device Applications , 2015, CHI.

[147]  Ulrich Eck,et al.  Precise Haptic Device Co-Location for Visuo-Haptic Augmented Reality , 2015, IEEE Transactions on Visualization and Computer Graphics.

[148]  Maristella Matera,et al.  Adaptive Interfaces , 2009, Encyclopedia of Database Systems.