Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR

Although Augmented Reality technology was first developed over forty years ago, there has been little survey work giving an overview of recent research in the field. This paper reviews the ten-year development of the work presented at the ISMAR conference and its predecessors with a particular focus on tracking, interaction and display research. It provides a roadmap for future augmented reality research which will be of great value to this relatively young field, and also for helping researchers decide which topics should be explored when they are beginning their own studies in the area.

[1]  Sean White,et al.  Visual Hints for Tangible Gestures in Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[2]  Mark Billinghurst,et al.  An occlusion capable optical see-through head mount display for supporting co-located collaboration , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[3]  Mark Billinghurst,et al.  Face to face collaborative AR on mobile phones , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[4]  Tom Drummond,et al.  Initialisation for Visual Tracking in Urban Environments , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[5]  Didier Stricker,et al.  Adaptive line tracking with multiple hypotheses for augmented reality , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[6]  Steven K. Feiner,et al.  Enveloping users and computers in a collaborative 3D augmented reality , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[7]  Steven K. Feiner,et al.  Immersive mixed-reality configuration of hybrid user interfaces , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[8]  Steven K. Feiner,et al.  A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[9]  Narendra Ahuja,et al.  Using a head-mounted projective display in interactive augmented environments , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[10]  Jeremiah Neubert,et al.  Semi-Autonomous Generation of Appearance-based Edge Models from Image Sequences , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[11]  Michael Harrington,et al.  FlightTracker: a novel optical/inertial tracker for cockpit enhanced vision , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[12]  三嶋 博之.  The theory of affordances , 2008 .

[13]  Dieter Schmalstieg,et al.  “Studierstube”: An environment for collaboration in augmented reality , 1998, Virtual Reality.

[14]  Anoop Gupta,et al.  Communication behaviors of co-located users in collaborative AR interfaces , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[15]  Dieter Schmalstieg,et al.  ARToolKit on the PocketPC platform , 2003, 2003 IEEE International Augmented Reality Toolkit Workshop.

[16]  Éric Marchand,et al.  Hybrid tracking algorithms for planar and non-planar structures subject to illumination changes , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[17]  Jakob Nielsen,et al.  Usability engineering , 1997, The Computer Science and Engineering Handbook.

[18]  Touradj Ebrahimi,et al.  Hybrid Tracking for Augmented Reality , 2005 .

[19]  Axel Pinz,et al.  Building a hybrid tracking system: integration of optical and magnetic tracking , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[20]  Gordon Wetzstein,et al.  Enabling view-dependent stereoscopic projection in real environments , 2005, SIGGRAPH '05.

[21]  Michitaka Hirose,et al.  Projected augmentation - augmented reality using rotatable video projectors , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[22]  Dieter Schmalstieg,et al.  Mobile collaborative augmented reality , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[23]  Ulrich Neumann,et al.  Dynamic registration correction in video-based augmented reality systems , 1995, IEEE Computer Graphics and Applications.

[24]  Henry Fuchs,et al.  Simulation-based design and rapid prototyping of a parallax-free, orthoscopic video see-through head-mounted display , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[25]  Nando de Freitas,et al.  Sequential Monte Carlo in Practice , 2001 .

[26]  Hiroshi Ishii,et al.  Triangles: tangible interface for manipulation and exploration of digital information topography , 1998, CHI.

[27]  Christopher O. Jaynes,et al.  The universal media book: tracking and augmenting moving surfaces with projected information , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[28]  Bruce H. Thomas,et al.  Implementation of god-like interaction techniques for supporting collaboration between outdoor AR and indoor tabletop users , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[29]  G. A. Giraldi,et al.  Introduction to Augmented Reality , 2003 .

[30]  Alexei A. Goon,et al.  A SURVEY OF TRACKING TECHNOLOGY FOR VIRTUAL ENVIRONMENTS , 1999 .

[31]  Heinrich Niemann,et al.  Single camera tracking of marker clusters: multiparameter cluster optimization and experimental verification , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[32]  Colin Potts,et al.  Design of Everyday Things , 1988 .

[33]  Pascal Fua,et al.  Vision Based 3D Tracking and Pose Estimation for Mixed Reality , 2007 .

[34]  G. Klinker,et al.  A fast and robust line-based optical tracker for augmented reality applications , 1999 .

[35]  G. Roth,et al.  Hand tracking for interactive pattern-based augmented reality , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[36]  Gordon Wetzstein,et al.  Consistent illumination within optical see-through augmented environments , 2003, SIGGRAPH '03.

[37]  Naokazu Yokoya,et al.  A stereo vision-based augmented reality system with an inertial sensor , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[38]  Alex Olwal,et al.  ASTOR: an autostereoscopic optical see-through augmented reality system , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[39]  Harald Ganster,et al.  Hybrid Tracking for Augmented Reality , 2000 .

[40]  Hideyuki Tamura,et al.  Robot vision-based registration utilizing bird's-eye view with user's view , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[41]  Dieter Schmalstieg,et al.  Experiences with Handheld Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[42]  Mark Weiser.  The computer for the 21st century , 1991 .

[43]  David G. Lowe,et al.  Scene modelling, recognition and tracking with invariant image features , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[44]  Tomohiro Tanikawa,et al.  Display Composed of Water Drops for Filling Space with Materialized Virtual Three-dimensional Objects , 2006, IEEE Virtual Reality Conference (VR 2006).

[45]  Gudrun Klinker,et al.  A System Architecture for Ubiquitous Tracking Environments , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[46]  Steven K. Feiner,et al.  Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system , 1999, Comput. Graph..

[47]  박영수,et al.  Optical system for head mounted display , 2006 .

[48]  Dieter Schmalstieg,et al.  Ubiquitous tracking for augmented reality , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[49]  Hirokazu Kato,et al.  Collaborative augmented reality , 2002, CACM.

[50]  Daniel Wagner,et al.  Augmented reality kanji learning , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[51]  Jannick P. Rolland,et al.  A Survey of Tracking Technologies for Virtual Environments , 2001 .

[52]  Eric Foxlin,et al.  Circular data matrix fiducial system and robust image processing for a wearable vision-inertial self-tracker , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[53]  G. Klein,et al.  Parallel Tracking and Mapping for Small AR Workspaces , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[54]  Kiyoshi Kiyokawa.  An Introduction to Head Mounted Displays for Augmented Reality , 2007 .

[55]  Ulrich Neumann,et al.  A multi-ring fiducial system and an intensity-invariant detection method for scalable augmented reality , 1999 .

[56]  Hiroshi Ishii,et al.  AR-Jig: A Handheld Tangible User Interface for Modification of 3D Digital Form via 2D Physical Curve , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[57]  Éric Marchand,et al.  A real-time tracker for markerless augmented reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[58]  Dieter Schmalstieg,et al.  Finger tracking for interaction in augmented environments , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[59]  Heedong Ko,et al.  "Move the couch where?" : developing an augmented reality multimodal interface , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[60]  Bernd Kleinjohann,et al.  Mobile AR4ALL , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[61]  Dieter Schmalstieg,et al.  Handheld Augmented Reality Displays , 2006, IEEE Virtual Reality Conference (VR 2006).

[62]  Mark Billinghurst,et al.  Shared Space: An Augmented Reality Interface for Computer Supported Collaborative Work , 1996 .

[63]  Oliver Bimber,et al.  Occlusion shadows: using projected light to generate realistic occlusion effects for view-dependent optical see-through displays , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[64]  Oliver Bimber,et al.  Video see-through AR on consumer cell-phones , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[65]  Bernd Brügge,et al.  Distributed user tracking concepts for augmented reality applications , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[66]  Lucy A. Suchman,et al.  Plans and Situated Actions: The Problem of Human-Machine Communication (Learning in Doing: Social, , 1987 .

[67]  Steven K. Feiner,et al.  Collaborative mixed reality visualization of an archaeological excavation , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[68]  David Ingram,et al.  Augmented reality in a wide area sentient environment , 2001, Proceedings IEEE and ACM International Symposium on Augmented Reality.

[69]  Sebastian Thrun,et al.  Probabilistic robotics , 2002, CACM.

[70]  Didier Stricker,et al.  Online camera pose estimation in partially known and dynamic scenes , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[71]  Tom Drummond,et al.  Robust visual tracking for non-instrumental augmented reality , 2003, The Second IEEE and ACM International Symposium on Mixed and Augmented Reality, 2003. Proceedings..

[72]  Sean White,et al.  Tangible progress: less is more in Somewire audio spaces , 1999, CHI '99.

[73]  Vincent Lepetit,et al.  Combining edge and texture information for real-time accurate 3D camera tracking , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[74]  Dieter Schmalstieg,et al.  Towards Massively Multi-user Augmented Reality on Handheld Devices , 2005, Pervasive.

[75]  Harald Ganster,et al.  Hybrid Tracking for Outdoor Augmented Reality Applications , 2002, IEEE Computer Graphics and Applications.

[76]  J. Rolland,et al.  Head-worn displays: a review , 2006, Journal of Display Technology.

[77]  Tom Drummond,et al.  Localisation and interaction for augmented maps , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[78]  Ronald Azuma,et al.  Making augmented reality work outdoors requires hybrid tracking , 1999 .

[79]  Bruce H. Thomas,et al.  Emerging technologies of augmented reality - interfaces and design , 2006 .

[80]  Ivan Poupyrev,et al.  Virtual object manipulation on a table-top AR environment , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[81]  John B. Shoven,et al.  I , Edinburgh Medical and Surgical Journal.

[82]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[83]  Dieter Schmalstieg,et al.  Bridging multiple user interface dimensions with augmented reality , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[84]  Jun Rekimoto,et al.  Matrix: a realtime object identification and registration method for augmented reality , 1998, Proceedings. 3rd Asia Pacific Computer Human Interaction (Cat. No.98EX110).

[85]  Hiroyuki Ohno,et al.  An optical see-through display for mutual occlusion of real and virtual environments , 2000, Proceedings IEEE and ACM International Symposium on Augmented Reality (ISAR 2000).

[86]  Nassir Navab,et al.  Visual marker detection and decoding in AR systems: a comparative study , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[87]  E. Tronci,et al.  1996 , 1997, Affair of the Heart.

[88]  Yakup Genc,et al.  Marker-less tracking for AR: a learning-based approach , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[89]  J P Rolland,et al.  Engineering of head-mounted projective displays. , 2000, Applied optics.

[90]  Eric Foxlin,et al.  Encoded LED system for optical trackers , 2005, Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05).

[91]  Tom Drummond,et al.  Going out: robust model-based tracking for outdoor augmented reality , 2006, 2006 IEEE/ACM International Symposium on Mixed and Augmented Reality.

[92]  Suya You,et al.  Natural feature tracking for extendible robust augmented realities , 1998 .

[93]  Ulrich Neumann,et al.  Vision-based pose computation: robust and accurate augmented reality tracking , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[94]  Hirokazu Kato,et al.  Marker tracking and HMD calibration for a video-based augmented reality conferencing system , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[95]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[96]  Markus H. Gross,et al.  Embedding imperceptible patterns into projected images for simultaneous acquisition and display , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.

[97]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[98]  Claudio S. Pinhanez.  The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces , 2001, UbiComp.

[99]  M. Weiser.  The Computer for the Twenty-First Century , 1991 .

[100]  Fumihisa Shibata,et al.  A Two-by-Two Mixed Reality System That Merges Real and Virtual Worlds in Both Audio and Visual Senses , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[101]  A. Pinz,et al.  Inertial tracking for mobile augmented reality , 2002, IMTC/2002. Proceedings of the 19th IEEE Instrumentation and Measurement Technology Conference (IEEE Cat. No.00CH37276).

[102]  Ivan E. Sutherland,et al.  The Ultimate Display , 1965 .

[103]  Ramesh Raskar,et al.  Session details: Course 17: Spatial augmented reality: merging real and virtual worlds , 2007, ACM SIGGRAPH 2007 courses.

[104]  Hiroyuki Yamamoto,et al.  A head tracking method using bird's-eye view camera and gyroscope , 2004, Third IEEE and ACM International Symposium on Mixed and Augmented Reality.