Design, implementation, and metrological characterization of a wearable, integrated AR-BCI hands-free system for health 4.0 monitoring

Abstract An integrated real-time monitoring system based on Augmented Reality (AR) and Brain-Computer Interface (BCI) for hands-free acquisition and visualization of remote data is proposed. As a case study, the monitoring of patients’ vitals in the operating room (OR) is considered; in particular, through the suitable combination of BCI and AR, the anesthetist can monitor in real-time (through a set of AR glasses), the patient’s vitals acquired from the electromedical equipment. Healthcare-related applications are particularly demanding in terms of real-time requirements; hence, the considered scenario represents an interesting and challenging testbed for the proposed system. Experimental tests were carried out at the University Hospital Federico II (Naples, Italy), employing pieces of equipment that are generally available in the OR. After the preliminary functional validation, accuracy and delay were measured, demonstrating the effectiveness and reliability of the proposed AR-BCI-based monitoring system.

[1]  L. D. De Paolis,et al.  Augmented visualization with depth perception cues to improve the surgeon’s performance in minimally invasive surgery , 2018, Medical & biological engineering & computing.

[2]  Pan Hui,et al.  Future Networking Challenges: The Case of Mobile Augmented Reality , 2017, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS).

[3]  Fabio Leccese,et al.  A step toward the standardization of maintenance and training services in C4I military systems with Mixed Reality application , 2019, Measurement.

[4]  Ediana Sutjiredjeki,et al.  A prototype of SSVEP-based BCI for home appliances control , 2016, 2016 1st International Conference on Biomedical Engineering (IBIOMED).

[5]  Kai Ma,et al.  A wristband device for detecting human pulse and motion based on the Internet of Things , 2020 .

[6]  Xiaorong Gao,et al.  Design and implementation of a brain-computer interface with high transfer rates , 2002, IEEE Transactions on Biomedical Engineering.

[7]  Pasquale Arpaia,et al.  A Single-Channel SSVEP-Based Instrument With Off-the-Shelf Components for Trainingless Brain-Computer Interfaces , 2019, IEEE Transactions on Instrumentation and Measurement.

[8]  Yang Yu,et al.  A Dynamically Optimized SSVEP Brain–Computer Interface (BCI) Speller , 2015, IEEE Transactions on Biomedical Engineering.

[9]  Luca T. Mainardi,et al.  Online Detection of P300 and Error Potentials in a BCI Speller , 2010, Comput. Intell. Neurosci..

[10]  Yijun Wang,et al.  Control of a 7-DOF Robotic Arm System With an SSVEP-Based BCI , 2018, Int. J. Neural Syst..

[11]  Syed Mahfuzul Aziz,et al.  Review of Cyber-Physical System in Healthcare , 2014, Int. J. Distributed Sens. Networks.

[12]  Pasquale Arpaia,et al.  Wearable Brain–Computer Interface Instrumentation for Robot-Based Rehabilitation by Augmented Reality , 2020, IEEE Transactions on Instrumentation and Measurement.

[13]  Nihat Daldal,et al.  Estimation of body fat percentage using hybrid machine learning algorithms , 2021, Measurement.

[14]  Gernot R. Müller-Putz,et al.  Control of an Electrical Prosthesis With an SSVEP-Based BCI , 2008, IEEE Transactions on Biomedical Engineering.

[15]  S. Nishtar,et al.  Health and healthcare in the Fourth Industrial Revolution: Global Future Council on the Future of Health and Healthcare 2016-2018 , 2019 .

[16]  Soung Chang Liew,et al.  A multiplexing scheme for H.323 voice-over-IP applications , 2002, IEEE J. Sel. Areas Commun..

[17]  Ayman Atia,et al.  Brain computer interfacing: Applications and challenges , 2015 .

[18]  D. Louis Collins,et al.  DVV: A Taxonomy for Mixed Reality Visualization in Image Guided Surgery , 2012, IEEE Transactions on Visualization and Computer Graphics.

[19]  S. Sokol,et al.  Visually evoked potentials: theory, techniques and clinical applications. , 1976, Survey of ophthalmology.

[20]  Piotr Milanowski,et al.  Towards an Optimization of Stimulus Parameters for Brain-Computer Interfaces Based on Steady State Visual Evoked Potentials , 2014, PloS one.

[21]  G. Monti,et al.  Dry Textile Electrodes for Wearable Bio-Impedance Analyzers , 2020, IEEE Sensors Journal.

[22]  Pasquale Arpaia,et al.  A Wearable Brain–Computer Interface Instrument for Augmented Reality-Based Inspection in Industry 4.0 , 2020, IEEE Transactions on Instrumentation and Measurement.

[23]  Ahmed M. Soliman,et al.  Analyzing patient health information based on IoT sensor with AI for improving patient assistance in the future direction , 2020 .

[24]  Athanasios V. Vasilakos,et al.  Brain computer interface: control signals review , 2017, Neurocomputing.

[25]  Álvaro Alesanco Iglesias,et al.  Clinical Assessment of Wireless ECG Transmission in Real-Time Cardiac Telemonitoring , 2010, IEEE Transactions on Information Technology in Biomedicine.

[26]  Robertas Damasevicius,et al.  A Prototype SSVEP Based Real Time BCI Gaming System , 2016, Comput. Intell. Neurosci..

[27]  Pasquale Arpaia,et al.  A Wearable EEG Instrument for Real-Time Frontal Asymmetry Monitoring in Worker Stress Analysis , 2020, IEEE Transactions on Instrumentation and Measurement.

[28]  Luciano Tarricone,et al.  Wearable Antennas: Nontextile Versus Fully Textile Solutions , 2019, IEEE Antennas and Propagation Magazine.

[29]  Anand Sánchez-Orta,et al.  On the Brain Computer Robot Interface (BCRI) to Control Robots , 2015, SyRoCo.

[30]  Julien Penders,et al.  Wearable, Wireless EEG Solutions in Daily Life Applications: What are we Missing? , 2015, IEEE Journal of Biomedical and Health Informatics.

[31]  Penelope M Sanderson,et al.  Advanced Auditory Displays and Head-Mounted Displays: Advantages and Disadvantages for Monitoring by the Distracted Anesthesiologist , 2008, Anesthesia and analgesia.

[32]  Torki A. Altameem,et al.  Internet of things sensor assisted security and quality analysis for health care data sets using artificial intelligent based heuristic health management system , 2020 .

[33]  Arianna Menciassi,et al.  Low-Computational Cost Stitching Method in a Three-Eyed Endoscope , 2019, Journal of healthcare engineering.

[34]  L. Carretié.  Exogenous (automatic) attention to emotional stimuli: a review , 2014, Cognitive, Affective, & Behavioral Neuroscience.

[35]  Amir A. Zadpoor,et al.  Additive Manufacturing of Biomaterials, Tissues, and Organs , 2016, Annals of Biomedical Engineering.

[36]  Dongmei Zhao,et al.  Providing telemedicine services in an infrastructure-based cognitive radio network , 2010, IEEE Wireless Communications.

[37]  Luciano Tarricone,et al.  Feasibility of a Wearable Reflectometric System for Sensing Skin Hydration , 2020, Sensors.

[38]  Heba M. Lakany,et al.  The Effect of the Viewing Distance of Stimulus on SSVEP Response for Use in Brain-Computer Interfaces , 2013, 2013 IEEE International Conference on Systems, Man, and Cybernetics.

[39]  Piotr Stawicki,et al.  Age-related differences in SSVEP-based BCI performance , 2017, Neurocomputing.

[40]  Shangkai Gao,et al.  A practical VEP-based brain-computer interface. , 2006, IEEE transactions on neural systems and rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[41]  Yu-Te Wang,et al.  A cell-phone-based brain–computer interface for communication in daily life , 2010, AICI.

[42]  G. Pfurtscheller,et al.  Brain-Computer Interfaces for Communication and Control. , 2011, Communications of the ACM.

[43]  Christophe Hurter,et al.  Pulse and vital sign measurement in mixed reality using a HoloLens , 2017, VRST.

[44]  Mark Wehde.  Healthcare 4.0 , 2019, IEEE Engineering Management Review.

[45]  Hamid Jahankhani,et al.  Health Care in the Cyberspace: Medical Cyber-Physical System and Digital Twin Challenges , 2019, Internet of Things.

[46]  E. Donchin,et al.  COGNITIVE PSYCHOPHYSIOLOGY: THE ENDOGENOUS COMPONENTS OF THE ERP , 1978 .

[47]  Annarita Tedesco,et al.  A preliminary discussion of measurement and networking issues in cyber physical systems for industrial manufacturing , 2017, 2017 IEEE International Workshop on Measurement and Networking (M&N).

[48]  Roland Siegwart,et al.  Article in Press Robotics and Autonomous Systems ( ) – Robotics and Autonomous Systems Brain-coupled Interaction for Semi-autonomous Navigation of an Assistive Robot , 2022 .

[49]  Alkinoos Athanasiou,et al.  Towards Brain-Computer Interface Control of a 6-Degree-of-Freedom Robotic Arm Using Dry EEG Electrodes , 2013, Adv. Hum. Comput. Interact..

[50]  Alaa Mohamed Riad,et al.  A machine learning model for improving healthcare services on cloud computing environment , 2018 .

[51]  Fotis Liarokapis,et al.  EEG-based BCI and video games: a progress report , 2018, Virtual Reality.

[52]  Dean J Krusienski,et al.  Brain-computer interface signal processing at the Wadsworth Center: mu and sensorimotor beta rhythms. , 2006, Progress in brain research.

[53]  D F Ormerod,et al.  Use of an augmented reality display of patient monitoring data to enhance anesthesiologists' response to abnormal clinical events. , 2003, Studies in health technology and informatics.

[54]  Rashid Mehmood,et al.  UbeHealth: A Personalized Ubiquitous Cloud and Edge-Enabled Networked Healthcare System for Smart Cities , 2018, IEEE Access.

[55]  Wendong Xu,et al.  Scientific profile of brain–computer interfaces: Bibliometric analysis in a 10-year period , 2016, Neuroscience Letters.

[56]  Amin Mahnam,et al.  Development of a practical high frequency brain–computer interface based on steady-state visual evoked potentials using a single channel of EEG , 2018 .

[57]  Felix Putze,et al.  Editorial: Brain-Computer Interfaces and Augmented/Virtual Reality , 2020, Frontiers in Human Neuroscience.

[58]  E Donchin,et al.  Brain-computer interface technology: a review of the first international meeting. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[59]  P. Arpaia,et al.  Metrology-Based Design of a Wearable Augmented Reality System for Monitoring Patient’s Vitals in Real Time , 2021, IEEE Sensors Journal.

[60]  Pasquale Arpaia,et al.  Wearable Augmented Reality and Brain Computer Interface to Improve Human-Robot Interactions in Smart Industry: A Feasibility Study for SSVEP Signals , 2018, 2018 IEEE 4th International Forum on Research and Technology for Society and Industry (RTSI).

[61]  Luc Soler,et al.  The status of augmented reality in laparoscopic surgery as of 2016 , 2017, Medical Image Anal..

[62]  Monica Bordegoni,et al.  Towards augmented reality manuals for industry 4.0: A methodology , 2019, Robotics and Computer-Integrated Manufacturing.

[63]  Hubert Cecotti,et al.  A Self-Paced and Calibration-Less SSVEP-Based Brain–Computer Interface Speller , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[64]  Lucio Tommaso De Paolis.  Augmented Visualization as Surgical Support in the Treatment of Tumors , 2017, IWBBIO.

[65]  Lorenzo Scalise,et al.  Wrist-worn and chest-strap wearable devices: Systematic review on accuracy and metrological characteristics , 2020 .