Interface of mixed reality: from the past to the future

Mixed reality (MR) is an emerging technology which could potentially shape the future of our everyday lives by its unique approach to presenting information. Technology is changing rapidly and information can be presented on traditional computer screens following a WIMP (Windows, Icons, Menus, and Pointing) interface model, by using a head-mounted display to present virtual reality, or by MR which the process of presenting information through a combination of both virtual and physical elements. This paper classifies MR interfaces by applying a text mining method to a data base of 4296 relevant research papers published over the last two decades. The classification reveals the trends relating to each topic and the relations between them. This paper reviews the earlier studies and discusses the recent developments in each topic area and summarizes the advantages and disadvantages of the MR interface. Our objective is to assist researchers understand the trend for each topic and allows them to focus on the research challenges where technological advancements in the MR interface are most needed.

[1]  Chih-Ming Chen,et al.  Interactive augmented reality system for enhancing library instruction in elementary schools , 2013, Comput. Educ..

[2]  Tony P. Pridmore,et al.  Expected, sensed, and desired: A framework for designing sensing-based interaction , 2005, TCHI.

[3]  Henry Been-Lirn Duh,et al.  Mixed Reality in Education: Recent Developments and Future Trends , 2018, 2018 IEEE 18th International Conference on Advanced Learning Technologies (ICALT).

[4]  Pan Hui,et al.  Ubii: Physical World Interaction Through Augmented Reality , 2017, IEEE Transactions on Mobile Computing.

[5]  Manfred Tscheligi,et al.  Augmented reality for industrial robot programmers: Workload analysis for task-based, augmented reality-supported robot control , 2016, 2016 25th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[6]  Hongwei Zhu,et al.  Recent advances in wearable tactile sensors: Materials, sensing mechanisms, and device performance , 2017 .

[7]  Steven K. Feiner,et al.  Exploring the Benefits of Augmented Reality Documentation for Maintenance and Repair , 2011, IEEE Transactions on Visualization and Computer Graphics.

[8]  Steve Benford,et al.  Where on-line meets on the streets: experiences with mobile mixed reality games , 2003, CHI '03.

[9]  Long Chen,et al.  Recent Developments and Future Challenges in Medical Mixed Reality , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[10]  Morten Fjeld,et al.  Tangible user interface for chemistry education: comparative evaluation and re-design , 2007, CHI.

[11]  J. Shuhaiber,et al.  Augmented reality in surgery. , 2004, Archives of surgery.

[12]  Kenneth R. Koedinger,et al.  Adding Physical Objects to an Interactive Game Improves Learning and Enjoyment , 2016, ACM Trans. Comput. Hum. Interact..

[13]  Hiroshi Ishii,et al.  Sensetable: a wireless object tracking platform for tangible user interfaces , 2001, CHI.

[14]  Mohan M. Trivedi,et al.  Head Pose Estimation and Augmented Reality Tracking: An Integrated System and Evaluation for Monitoring Driver Awareness , 2010, IEEE Transactions on Intelligent Transportation Systems.

[15]  Daniel Sonntag,et al.  ModulAR: Eye-Controlled Vision Augmentations for Head Mounted Displays , 2015, IEEE Transactions on Visualization and Computer Graphics.

[16]  Kenneth R. Koedinger,et al.  Learning from Mixed-Reality Games: Is Shaking a Tablet as Effective as Physical Observation? , 2015, CHI.

[17]  Alexandre Gillet,et al.  Tangible interfaces for structural molecular biology. , 2005, Structure.

[18]  Brett E. Shelton,et al.  Using augmented reality for teaching Earth-Sun relationships to undergraduate geography students , 2002, The First IEEE International Workshop Agumented Reality Toolkit,.

[19]  Jovan Popović,et al.  Real-time hand-tracking with a color glove , 2009, SIGGRAPH 2009.

[20]  Carlo Alberto Avizzano,et al.  Immersive ROS-integrated framework for robot teleoperation , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[21]  Éric Marchand,et al.  Real-time markerless tracking for augmented reality: the virtual visual servoing framework , 2006, IEEE Transactions on Visualization and Computer Graphics.

[22]  J. Ferrer-Torregrosa,et al.  Distance learning ects and flipped classroom in the anatomy learning: comparative study of the use of augmented reality, video and notes , 2016, BMC Medical Education.

[23]  Emi Tamaki,et al.  UnlimitedHand: Input and Output Hand Gestures with Less Calibration Time , 2016, UIST.

[24]  Guangyong Li,et al.  "Videolized" atomic force microscopy for interactive nanomanipulation and nanoassembly , 2005, IEEE Transactions on Nanotechnology.

[25]  Naokazu Yokoya,et al.  A stereoscopic video see-through augmented reality system based on real-time vision-based registration , 2000, Proceedings IEEE Virtual Reality 2000 (Cat. No.00CB37048).

[26]  Gudrun Klinker,et al.  Tangible Mixed Reality On-Site: Interactive Augmented Visualisations from Architectural Working Models in Urban Design , 2015 .

[27]  Steve Benford,et al.  The Frame of the Game: Blurring the Boundary between Fiction and Reality in Mobile Experiences , 2006, CHI 2006.

[28]  Shahzad Malik,et al.  Visual touchpad: a two-handed gestural input device , 2004, ICMI '04.

[29]  Gregory D. Abowd,et al.  Charting past, present, and future research in ubiquitous computing , 2000, TCHI.

[30]  Hiroshi Ishii,et al.  Tangible bits: beyond pixels , 2008, TEI.

[31]  S. Botden,et al.  Augmented versus Virtual Reality Laparoscopic Simulation: What Is the Difference? , 2007, World journal of surgery.

[32]  J. Marescaux,et al.  Telerobotic Laparoscopic Cholecystectomy: Initial Clinical Experience With 25 Patients , 2001, Annals of surgery.

[33]  Steven K. Feiner,et al.  A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[34]  Steven K. Feiner,et al.  Enveloping users and computers in a collaborative 3D augmented reality , 1999, Proceedings 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99).

[35]  Jan Egger,et al.  Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display , 2015, J. Biomed. Informatics.

[36]  Ning Xi,et al.  Development of augmented reality system for AFM-based nanomanipulation , 2004 .

[37]  Frances M. T. Brazier,et al.  On the Usability and Effectiveness of Different Interaction Types in Augmented Reality , 2015, Int. J. Hum. Comput. Interact..

[38]  Patrick Baudisch,et al.  Lucid touch: a see-through mobile device , 2007, UIST.

[39]  Youchang Kim,et al.  18.1 A 2.71nJ/pixel 3D-stacked gaze-activated object-recognition system for low-power mobile HMD applications , 2015, 2015 IEEE International Solid-State Circuits Conference - (ISSCC) Digest of Technical Papers.

[40]  Voicu Popescu,et al.  Medical telementoring using an augmented reality transparent display. , 2016, Surgery.

[41]  S. Botden,et al.  What is going on in augmented reality simulation in laparoscopic surgery? , 2008, Surgical Endoscopy.

[42]  Neil Gordon,et al.  Heuristic Evaluation for Serious Immersive Games and M-instruction , 2016, HCI.

[43]  Hiroshi Ishii,et al.  Tangible bits: towards seamless interfaces between people, bits and atoms , 1997, CHI.

[44]  E. Noé,et al.  Effect of a mixed reality-based intervention on arm, hand, and finger function on chronic stroke , 2016, Journal of NeuroEngineering and Rehabilitation.

[45]  Wolfgang Birkfellner,et al.  A head-mounted operating binocular for augmented reality visualization in medicine - design and initial evaluation , 2002, IEEE Transactions on Medical Imaging.

[46]  Nolan Ung,et al.  Utilizing virtual and augmented reality for educational and clinical enhancements in neurosurgery , 2017, Journal of Clinical Neuroscience.

[47]  Jun Rekimoto,et al.  Augmented surfaces: a spatially continuous work space for hybrid computing environments , 1999, CHI '99.

[48]  Nazim Haouchine,et al.  Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[49]  Leila Besharati Tabrizi,et al.  Augmented reality-guided neurosurgery: accuracy and intraoperative application of an image projection technique. , 2015, Journal of neurosurgery.

[50]  Dieter Schmalstieg,et al.  The Studierstube Augmented Reality Project , 2002, Presence: Teleoperators & Virtual Environments.

[51]  Tobias Höllerer,et al.  Interpreting 2D gesture annotations in 3D augmented reality , 2016, 2016 IEEE Symposium on 3D User Interfaces (3DUI).

[52]  Ajune Wanis Ismail,et al.  Multimodal Fusion: Gesture and Speech Input in Augmented Reality Environment , 2014, INNS-CIIS.

[53]  Pattie Maes,et al.  WUW - wear Ur world: a wearable gestural interface , 2009, CHI Extended Abstracts.

[54]  R. Sweet The CREST Simulation Development Process: Training the Next Generation. , 2017, Journal of endourology.

[55]  M. Schijven,et al.  Systematic review on the effectiveness of augmented reality applications in medical training , 2016, Surgical Endoscopy.

[56]  Thomas B. Moeslund,et al.  A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI , 2003, Gesture Workshop.

[57]  Hrvoje Benko,et al.  Combining multiple depth cameras and projectors for interactions on, above and between surfaces , 2010, UIST.

[58]  Robert J. K. Jacob,et al.  What you look at is what you get , 2016, Interactions.

[59]  Nassir Navab,et al.  Contextual Anatomic Mimesis Hybrid In-Situ Visualization Method for Improving Multi-Sensory Depth Perception in Medical Augmented Reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[60]  Ichiro Sakuma,et al.  Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation , 2015, Comput. Medical Imaging Graph..

[61]  Pan Hui,et al.  Ubii: Towards Seamless Interaction between Digital and Physical Worlds , 2015, ACM Multimedia.

[62]  Morten Fjeld,et al.  Physical and Virtual Tools: Activity Theory Applied to the Design of Groupware , 2002, Computer Supported Cooperative Work (CSCW).

[63]  Anoop Gupta,et al.  Communication behaviors of co-located users in collaborative AR interfaces , 2002, Proceedings. International Symposium on Mixed and Augmented Reality.

[64]  Paolo Dario,et al.  EndoCAS navigator platform: a common platform for computer and robotic assistance in minimally invasive surgery , 2008, The international journal of medical robotics + computer assisted surgery : MRCAS.

[65]  Steven K. Feiner,et al.  View management for virtual and augmented reality , 2001, UIST '01.

[66]  Steven K. Feiner,et al.  Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system , 1999, Comput. Graph..

[67]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[68]  N. Brandt,et al.  Implementing Smart Urban Metabolism in the Stockholm Royal Seaport: Smart City SRS , 2015 .

[69]  Luca Paolino,et al.  Training emergency responders through augmented reality mobile interfaces , 2015, Multimedia Tools and Applications.

[70]  George Papagiannakis,et al.  A survey of mobile and wireless technologies for augmented reality systems , 2008 .

[71]  Chang-Hwan Im,et al.  Real-Time “Eye-Writing” Recognition Using Electrooculogram , 2017, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[72]  Hisayuki Tatsumi,et al.  Cane Walk in the Virtual Reality Space Using Virtual Haptic Sensing: Toward Developing Haptic VR Technologies for the Visually Impaired , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[73]  Yoichi Sato,et al.  Integrating paper and digital information on EnhancedDesk: a method for realtime finger tracking on an augmented desk system , 2001, TCHI.

[74]  Chris Dede,et al.  Affordances and Limitations of Immersive Participatory Augmented Reality Simulations for Teaching and Learning , 2009 .

[75]  Damianos Gavalas,et al.  SoundPacman: Audio augmented reality in location-based games , 2016, 2016 18th Mediterranean Electrotechnical Conference (MELECON).

[76]  Ken Masamune,et al.  An effective visualization technique for depth perception in augmented reality‐based surgical navigation , 2016, The international journal of medical robotics + computer assisted surgery : MRCAS.

[77]  David S. Goodsell,et al.  Augmented reality with tangible auto-fabricated models for molecular biology applications , 2004, IEEE Visualization 2004.

[78]  Kiriakos N. Kutulakos,et al.  Calibration-Free Augmented Reality , 1998, IEEE Trans. Vis. Comput. Graph..

[79]  Shuai Wang,et al.  Enhancing learning and engagement through embodied interaction within a mixed reality simulation , 2016, Comput. Educ..

[80]  Hans Peter Luhn,et al.  A Statistical Approach to Mechanized Encoding and Searching of Literary Information , 1957, IBM J. Res. Dev..

[81]  Joern Ploennigs,et al.  Materializing the Promises of Cognitive IoT: How Cognitive Buildings Are Shaping the Way , 2018, IEEE Internet of Things Journal.

[82]  Massimo Bergamasco,et al.  Evaluating the Impact of Highly Immersive Technologies and Natural Interaction on Player Engagement and Flow Experience in Games , 2015, ICEC.

[83]  Martin White,et al.  Exploring the relationship between presence and enjoyment in a virtual museum , 2010, Int. J. Hum. Comput. Stud..

[84]  Maria Frucci,et al.  Touchless Target Selection Techniques for Wearable Augmented Reality Systems , 2015 .

[85]  G. Appelboom,et al.  Clinical and surgical applications of smart glasses. , 2015, Technology and health care : official journal of the European Society for Engineering and Medicine.

[86]  Moncef L. Nehdi,et al.  Data acquisition technologies for construction progress tracking , 2016 .

[87]  Mohamed Abdur Rahman,et al.  A Multi-Sensory Gesture-Based Occupational Therapy Environment for Controlling Home Appliances , 2015, ICMR.

[88]  Eelke Folmer,et al.  Immersive Simulation of Visual Impairments Using a Wearable See-through Display , 2015, TEI.

[89]  Blair MacIntyre,et al.  DART: a toolkit for rapid design exploration of augmented reality experiences , 2004, UIST '04.

[90]  Jörg Krüger,et al.  Movement visualisation in virtual reality rehabilitation of the lower limb: a systematic review , 2016, BioMedical Engineering OnLine.

[91]  Guilherme C. S. Ruppert,et al.  Touchless gesture user interface for interactive image visualization in urological surgery , 2012, World Journal of Urology.

[92]  Jennifer Healey,et al.  Augmented Reality through Wearable Computing , 1997, Presence: Teleoperators & Virtual Environments.

[93]  Matthew Christopher Davis,et al.  Virtual Interactive Presence in Global Surgical Education: International Collaboration Through Augmented Reality. , 2016, World neurosurgery.

[94]  Anup Kumar,et al.  Current status of robotic simulators in acquisition of robotic surgical skills , 2015, Current opinion in urology.

[95]  Min Zheng,et al.  ChemPreview: an augmented reality-based molecular interface. , 2017, Journal of molecular graphics & modelling.

[96]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[97]  Holger Regenbrecht,et al.  Towards Pervasive Augmented Reality: Context-Awareness in Augmented Reality , 2017, IEEE Transactions on Visualization and Computer Graphics.

[98]  H Iseki,et al.  Volumegraph (overlaid three-dimensional image-guided navigation). Clinical application of augmented reality in neurosurgery. , 1997, Stereotactic and functional neurosurgery.

[99]  Sevda Küçük,et al.  Learning anatomy via mobile augmented reality: Effects on achievement and cognitive load , 2016, Anatomical sciences education.

[100]  Hoi-Jun Yoo,et al.  14.1 A 126.1mW real-time natural UI/UX processor with embedded deep-learning core for low-power smart glasses , 2016, 2016 IEEE International Solid-State Circuits Conference (ISSCC).

[101]  Gerhard Schwabe,et al.  Mobile learning with a mobile game: design and motivational effects , 2005, J. Comput. Assist. Learn..

[102]  Ernesto Damiani,et al.  Augmented reality technologies, systems and applications , 2010, Multimedia Tools and Applications.

[103]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[104]  Mark Billinghurst,et al.  Augmenting the science centre and museum experience , 2004, GRAPHITE '04.

[105]  Bruce H. Thomas,et al.  ARQuake: an outdoor/indoor augmented reality first person application , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[106]  Darwin G. Caldwell,et al.  Integrating Haptics with Augmented Reality in a Femoral Palpation and Needle Insertion Training Simulation , 2011, IEEE Transactions on Haptics.

[107]  Michel Beaudouin-Lafon,et al.  Instrumental interaction: an interaction model for designing post-WIMP user interfaces , 2000, CHI.

[108]  Eva Lindh Waterworth,et al.  Focus, Locus, and Sensus: The Three Dimensions of Virtual Experience , 2001, Cyberpsychology Behav. Soc. Netw..

[109]  Avikarsha Mandal,et al.  Possible applications of the LEAP motion controller for more interactive simulated experiments in augmented or virtual reality , 2016, Optical Engineering + Applications.

[110]  Bruce H. Thomas,et al.  Tinmith-Metro: new outdoor techniques for creating city models with an augmented reality wearable computer , 2001, Proceedings Fifth International Symposium on Wearable Computers.

[111]  Mark Billinghurst,et al.  The Effect of View Independence in a Collaborative AR System , 2015, Computer Supported Cooperative Work (CSCW).

[112]  Fengfeng Ke,et al.  Teaching training in a mixed-reality integrated learning environment , 2016, Comput. Hum. Behav..

[113]  A. Y. C. Nee,et al.  Multi-modal augmented-reality assembly guidance based on bare-hand interface , 2016, Adv. Eng. Informatics.

[114]  Kaisa Väänänen,et al.  Suitability of virtual prototypes to support human factors/ergonomics evaluation during the design. , 2016, Applied ergonomics.

[115]  Andy Cockburn,et al.  FingARtips: gesture based direct manipulation in Augmented Reality , 2004, GRAPHITE '04.

[116]  Adam Hamrol,et al.  Immersive City Bus Configuration System for Marketing and Sales Education , 2015 .

[117]  K. Schaller,et al.  Augmented reality-assisted bypass surgery: embracing minimal invasiveness. , 2015, World neurosurgery.

[118]  Nassir Navab,et al.  Personalized augmented reality for anatomy education , 2015, Clinical anatomy.

[119]  Hiroshi Ishii,et al.  Emerging frameworks for tangible user interfaces , 2000, IBM Syst. J..

[120]  William S. Harwin,et al.  Upper Limb Robot Mediated Stroke Therapy—GENTLE/s Approach , 2003, Auton. Robots.

[121]  Daniel J. Wigdor,et al.  Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality , 2016, CHI.

[122]  Ivan Poupyrev,et al.  The MagicBook: a transitional AR interface , 2001, Comput. Graph..

[123]  Terry M. Peters,et al.  Training for Planning Tumour Resection: Augmented Reality and Human Factors , 2015, IEEE Transactions on Biomedical Engineering.

[124]  Yoshifumi Kitamura,et al.  IM6D: magnetic tracking system with 6-DOF passive markers for dexterous 3D interaction and motion , 2015, ACM Trans. Graph..

[125]  Jun Rekimoto,et al.  CyberCode: designing augmented reality environments with visual tags , 2000, DARE '00.

[126]  Gilberto Osorio-Gómez,et al.  AIR-MODELLING: A tool for gesture-based solid modelling in context during early design stages in AR environments , 2015, Comput. Ind..

[127]  Bernd Girod,et al.  A Framework to Evaluate Omnidirectional Video Coding Schemes , 2015, 2015 IEEE International Symposium on Mixed and Augmented Reality.

[128]  Jürgen Jasperneite,et al.  A context-aware assistance system for maintenance applications in smart factories based on augmented reality and indoor localization , 2015, 2015 IEEE 20th Conference on Emerging Technologies & Factory Automation (ETFA).

[129]  Eyal Ofek,et al.  Room2Room: Enabling Life-Size Telepresence in a Projected Augmented Reality Environment , 2016, CSCW.

[130]  Roy Want,et al.  Bridging physical and virtual worlds with electronic tags , 1999, CHI '99.

[131]  Imre J. Rudas,et al.  Design, programming and orchestration of heterogeneous manufacturing systems through VR-powered remote collaboration , 2015 .

[132]  Robert W. Lindeman,et al.  A Systematic Review of Usability Studies in Augmented Reality between 2005 and 2014 , 2016, 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct).

[133]  David E. Breen,et al.  Real-time vision-based camera tracking for augmented reality applications , 1997, VRST '97.

[134]  Wojciech Cellary,et al.  Evaluation of learners' attitude toward learning in ARIES augmented reality environments , 2013, Comput. Educ..

[135]  Joseph A. Paradiso,et al.  PingPongPlus: design of an athletic-tangible interface for computer-supported cooperative play , 1999, CHI '99.

[136]  P. Spiess,et al.  Augmented reality assisted surgery: a urologic training tool , 2015, Asian journal of andrology.

[137]  Mark Weiser The computer for the 21st century , 1991 .

[138]  Woontack Woo,et al.  3D Finger CAPE: Clicking Action and Position Estimation under Self-Occlusions in Egocentric Viewpoint , 2015, IEEE Transactions on Visualization and Computer Graphics.

[139]  Hirokazu Kato,et al.  Collaborative augmented reality , 2002, CACM.

[140]  Soh-Khim Ong,et al.  Real-virtual components interaction for assembly simulation and planning , 2016 .

[141]  Gudrun Klinker,et al.  Semi-Parametric Color Reproduction Method for Optical See-Through Head-Mounted Displays , 2015, IEEE Transactions on Visualization and Computer Graphics.

[142]  Ismo Rakkolainen,et al.  Head-mounted display with mid-air tactile feedback , 2015, VRST.

[143]  Ivan Poupyrev,et al.  The MagicBook - Moving Seamlessly between Reality and Virtuality , 2001, IEEE Computer Graphics and Applications.

[144]  Scott R. Klemmer,et al.  Papier-Mache: toolkit support for tangible input , 2004, CHI.

[145]  Bruno Simões,et al.  Creativity Support in Projection-Based Augmented Environments , 2015, AVR.

[146]  Michael J. Daly,et al.  Surgeons blinded by enhanced navigation: the effect of augmented reality on attention , 2013, Surgical Endoscopy.

[147]  Raffaele Di Fuccio,et al.  Towards Hyper Activity Books for Children. Connecting Activity Books and Montessori-like Educational Materials , 2015, EC-TEL.

[148]  Pedro F. Campos,et al.  SMART: a SysteM of Augmented Reality for Teaching 2 nd grade students , 2008 .

[149]  Tobias Höllerer,et al.  Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[150]  Rosemary Luckin,et al.  “Making it real”: exploring the potential of augmented reality for teaching primary school science , 2006, Virtual Reality.