A systematic literature review on dynamic cognitive augmentation through immersive reality: challenges and perspectives

Rapid advances in immersive reality technologies have resulted in a vast quantity of research papers which generally include or attempt to apply it to human cognitive augmentation. Taking advantage of traditional psychological and physiological metrics for studying human cognition and behavior, researchers are applying various near-real time analysis techniques to modulate immersive experiences and influence the mental state of the user. Because of the variety of contributing sub-domains, there is little consensus as to any rigid paradigm for the knowledge being synthesized. In this paper, we conduct a systematic literature review to determine the state of the art of dynamic cognitive augmentation in immersive environments. Following a structured query of academic publications, we conduct an in-depth analysis of 104 papers from a sample of 538. We observed that roughly 66% of papers among this frontier apply methods best suited for exploratory purposes, limiting the overall extent to which conclusions can be drawn about immersive reality technology’s capability to augment human cognition. We further identify a pressing gap in the knowledge necessary for the effective application of immersive reality towards dynamic cognitive augmentation in practical industrial scenarios. We hope this work will influence academia, industry, and standards development organizations to extend the use of XR technology networked with biosensor-enabled intelligent cognitive assistants to enhance the effectiveness of hybrid human-machine systems.

[1]  Yang Wang,et al.  Robust Multimodal Cognitive Load Measurement , 2016, Human–Computer Interaction Series.

[2]  Ulrich Neumann,et al.  Cognitive, performance, and systems issues for augmented reality applications in manufacturing and maintenance , 1998, Proceedings. IEEE 1998 Virtual Reality Annual International Symposium (Cat. No.98CB36180).

[3]  Javier Hernandez,et al.  SenseGlass: using google glass to sense daily emotions , 2014, UIST.

[4]  Ross T. Smith,et al.  Cognitive Cost of Using Augmented Reality Displays , 2017, IEEE Transactions on Visualization and Computer Graphics.

[5]  Leonardo Frizziero,et al.  Augmented Reality for virtual user manual , 2018 .

[6]  Yu Yuan,et al.  All Reality: Virtual, Augmented, Mixed (X), Mediated (X, Y), and Multimediated Reality , 2018, ArXiv.

[7]  Wolfgang Weller,et al.  Auf dem Weg zur 4. Industriellen Revolution , 2014 .

[8]  D. Tranfield,et al.  Towards a Methodology for Developing Evidence-Informed Management Knowledge by Means of Systematic Review , 2003 .

[9]  Roderich Groß,et al.  Using Google Glass in Human-Robot Swarm Interaction , 2016, TAROS.

[10]  Alasdair Gilchrist.  Industry 4.0: The Industrial Internet of Things , 2016 .

[11]  Steven E. Butt,et al.  Measuring mental workload in a hospital unit using EEG - A pilot study , 2014 .

[12]  F. Herrmann,et al.  Assessment of mental workload: A new electrophysiological method based on intra-block averaging of ERP amplitudes , 2016, Neuropsychologia.

[13]  Beth Coleman,et al.  Using Sensor Inputs to Affect Virtual and Real Environments , 2009, IEEE Pervasive Computing.

[14]  Bruno Simões,et al.  Technologies for Industry 4.0 Data Solutions , 2019 .

[15]  Álvaro Segura,et al.  Visual computing technologies to support the Operator 4.0 , 2020, Comput. Ind. Eng..

[16]  A. Santanaa,et al.  Rethinking Human-Machine Learning in Industry 4 . 0 : How Does the Paradigm Shift Treat the Role of Human Learning ? , 2018 .

[17]  R. Satava,et al.  Virtual Reality Training Improves Operating Room Performance: Results of a Randomized, Double-Blinded Study , 2002, Annals of surgery.

[18]  Marc H. Schieber,et al.  Advancing brain-machine interfaces: moving beyond linear state space models , 2015, Front. Syst. Neurosci..

[19]  Richard E. Mayer,et al.  Multimedia Learning: The Promise of Multimedia Learning , 2001 .

[20]  Brain Computer Interfaces, Principles and Practise , 2013 .

[21]  Barbara Kitchenham,et al.  Procedures for Performing Systematic Reviews , 2004 .

[22]  Jing Wang,et al.  A Brain-to-Brain Interface for Real-Time Sharing of Sensorimotor Information , 2013, Scientific Reports.

[23]  Ying Liu,et al.  A categorical framework of manufacturing for industry 4.0 and beyond , 2016 .

[24]  Steffen Staab,et al.  Web Science , 2013, Informatik-Spektrum.

[25]  Bruno Simões,et al.  Cross reality to enhance worker cognition in industrial assembly operations , 2019, The International Journal of Advanced Manufacturing Technology.

[26]  Riccardo Poli,et al.  Neurotechnologies for Human Cognitive Augmentation: Current State of the Art and Future Prospects , 2019, Front. Hum. Neurosci..

[27]  A. White,et al.  Systematic literature reviews. , 2005, Complementary therapies in medicine.

[28]  Anthony Steed,et al.  The impact of a self-avatar on cognitive load in immersive virtual reality , 2016, 2016 IEEE Virtual Reality (VR).

[29]  Denis Gracanin,et al.  Immersion Versus Embodiment: Embodied Cognition for Immersive Analytics in Mixed Reality Environments , 2018, HCI.

[30]  Anja S. Göritz,et al.  Virtual reality in the application of heart rate variability biofeedback , 2019, Int. J. Hum. Comput. Stud..

[31]  Åsa Fast-Berglund,et al.  Testing and validating Extended Reality (xR) technologies in manufacturing , 2018 .

[32]  Bruce H. Thomas,et al.  Improving procedural task performance with Augmented Reality annotations , 2013, 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).

[33]  W. Barfield,et al.  Cyborgs and Enhancement Technology , 2017 .

[34]  Joseph A. Paradiso,et al.  Guest Editors' Introduction: Cross-Reality Environments , 2009, IEEE Pervasive Comput..

[35]  Christoffer Rybski,et al.  Learning Factory for Industry 4.0 to provide future skills beyond technical training , 2018 .

[36]  Bruno Simões,et al.  X-Reality System Architecture for Industry 4.0 Processes , 2018, Multimodal Technol. Interact..

[37]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..