Brain-computer interface for hands-free teleoperation of construction robots

Abstract Recently, the use of collaborative robots has started to emerge at construction sites. Such incorporation into these human-dominated environments can raise safety concerns as most robots are not fully automated and require some sort of control mechanism. Conventional control systems may fall short under specific situations in which workers require robotic assistance but cannot use their hands to control the robot. Brain-computer interface (BCI) can offer such hands-free controllability, a non-muscular communicative channel that can establish an interpretive pathway between humans and robots. This paper proposes a BCI-based system to remotely control a robot by continuously capturing workers' brainwaves acquired from a wearable electroencephalogram (EEG) device and interpreting them into robotic commands with 90% accuracy. The findings revealed the proposed system holds promise for enhancing robot control in hazardous operations where the ability of the worker to physically direct the robot is limited, such as underwater and space construction.

[1]  Sungkon Moon,et al.  Discrete Firefly Algorithm for Scaffolding Construction Scheduling , 2017 .

[2]  V. Reggie Edgerton,et al.  Enhancing Nervous System Recovery through Neurobiologics, Neural Interface Training, and Neurorehabilitation , 2016, Front. Neurosci..

[3]  Wei Yan,et al.  A leader-follower formation control strategy for AUVs based on line-of-sight guidance , 2009, 2009 International Conference on Mechatronics and Automation.

[4]  SangHyun Lee,et al.  Wearable Sensing Technology Applications in Construction Safety and Health , 2019, Journal of Construction Engineering and Management.

[5]  Burcin Becerik-Gerber,et al.  Monitoring fatigue in construction workers using physiological measurements , 2017 .

[6]  Yang Li,et al.  Automatic Identification and Removal of Ocular Artifacts in EEG—Improved Adaptive Predictor Filtering for Portable Applications , 2014, IEEE Transactions on NanoBioscience.

[7]  Cuntai Guan,et al.  Filter Bank Common Spatial Pattern (FBCSP) in Brain-Computer Interface , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[8]  Tibor Ujbanyi,et al.  Speed control of Festo Robotino mobile robot using NeuroSky MindWave EEG headset based brain-computer interface , 2016, 2016 7th IEEE International Conference on Cognitive Infocommunications (CogInfoCom).

[9]  Cuntai Guan,et al.  Filter Bank Common Spatial Pattern Algorithm on BCI Competition IV Datasets 2a and 2b , 2012, Front. Neurosci..

[10]  Houtan Jebelli,et al.  Multi-Level Assessment of Occupational Stress in the Field Using a Wearable EEG Headset , 2020 .

[11]  K.-R. Muller,et al.  Optimizing Spatial filters for Robust EEG Single-Trial Analysis , 2008, IEEE Signal Processing Magazine.

[12]  Houtan Jebelli,et al.  Ocular Artifacts Reduction in EEG Signals Acquired at Construction Sites by Applying a Dependent Component Analysis (DCA) , 2020 .

[13]  Mihaly Benda,et al.  Brain–Computer Interface Spellers: A Review , 2018, Brain sciences.

[14]  Jonathan Kofman,et al.  Teleoperation of a robot manipulator using a vision-based human-robot interface , 2005, IEEE Transactions on Industrial Electronics.

[15]  Zoubeir Lafhaj,et al.  Robotic mechanical design for brick-laying automation , 2017 .

[16]  Jie Li,et al.  Motor imagery BCI-based robot arm system , 2011, 2011 Seventh International Conference on Natural Computation.

[17]  Andrzej Cichocki,et al.  Brain-Computer Interface with Corrupted EEG Data: a Tensor Completion Approach , 2018, Cognitive Computation.

[18]  Sotiris Makris,et al.  Augmented reality system for operator support in human–robot collaborative assembly , 2016 .

[19]  G. Pfurtscheller,et al.  Self-Paced Operation of an SSVEP-Based Orthosis With and Without an Imagery-Based “Brain Switch:” A Feasibility Study Towards a Hybrid BCI , 2010, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[20]  Piotr Stawicki,et al.  Driving a Semiautonomous Mobile Robotic Car Controlled by an SSVEP-Based BCI , 2016, Comput. Intell. Neurosci..

[21]  D M Durand,et al.  Suppression of axonal conduction by sinusoidal stimulation in rat hippocampus in vitro , 2007, Journal of neural engineering.

[22]  Dieter Schwarzmann,et al.  Control Concept for a Hydraulic Mobile Machine Using a Haptic Operating Device , 2009, 2009 Second International Conferences on Advances in Computer-Human Interactions.

[23]  Siddhartha S. Srinivasa,et al.  Effects of Robot Motion on Human-Robot Collaboration , 2015, 2015 10th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[24]  Ian Flood,et al.  The Need for Remote Artificial Intelligence Control of Space-Based Construction Projects: Multi-Agent-Based Last Planners, Local Centralized Controllers, and Hybrid Solutions to Decision-Making , 2018 .

[25]  Jonathan R. Wolpaw,et al.  Brain–Computer InterfacesPrinciples and Practice , 2012 .

[26]  Kyuwan Choi,et al.  Control of a Wheelchair by Motor Imagery in Real Time , 2008, IDEAL.

[27]  Houtan Jebelli,et al.  Measuring Workers’ Emotional State during Construction Tasks Using Wearable EEG , 2018, Journal of Construction Engineering and Management.

[28]  Myung Hwangbo,et al.  A stable target-tracking control for unicycle mobile robots , 2000, Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2000) (Cat. No.00CH37113).

[29]  David T. Branson,et al.  Assessing worker performance using dynamic cost functions in human robot collaborative tasks , 2020 .

[30]  Cuntai Guan,et al.  Multi-class filter bank common spatial pattern for four-class motor imagery BCI , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[31]  Chang-Hwan Im,et al.  EEG-Based Brain-Computer Interfaces: A Thorough Literature Survey , 2013, Int. J. Hum. Comput. Interact..

[32]  G. Pfurtscheller,et al.  Designing optimal spatial filters for single-trial EEG classification in a movement task , 1999, Clinical Neurophysiology.

[33]  Genshe Chen,et al.  Progress in EEG-Based Brain Robot Interaction Systems , 2017, Comput. Intell. Neurosci..

[34]  Lorenzo Beretta,et al.  Nearest neighbor imputation algorithms: a critical evaluation , 2016, BMC Medical Informatics and Decision Making.

[35]  Timothy Bretl,et al.  An Interactive Model-Driven Path Planning and Data Capture System for Camera-Equipped Aerial Robots on Construction Sites , 2017 .

[36]  Shinji Umeyama,et al.  An Eigendecomposition Approach to Weighted Graph Matching Problems , 1988, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Fabien Lotte,et al.  Brain-Computer Interfaces: Beyond Medical Applications , 2012, Computer.

[38]  André Crosnier,et al.  Collaborative manufacturing with physical human–robot interaction , 2016 .

[39]  Jaeseung Jeong,et al.  Brain-actuated humanoid robot navigation control using asynchronous Brain-Computer Interface , 2011, 2011 5th International IEEE/EMBS Conference on Neural Engineering.

[40]  J. R. Llata,et al.  Working Together: A Review on Safe Human-Robot Collaboration in Industrial Environments , 2017, IEEE Access.

[41]  Houtan Jebelli,et al.  EEG-based workers' stress recognition at construction sites , 2018, Automation in Construction.

[42]  Rihab Bousseta,et al.  EEG Based Brain Computer Interface for Controlling a Robot Arm Movement Through Thought , 2018 .

[43]  Sotiris Makris,et al.  Human–robot interaction review and challenges on task planning and programming , 2016, Int. J. Comput. Integr. Manuf..

[44]  Georges Bastin,et al.  Control of Nonholonomic Wheeled Mobile Robots by State Feedback Linearization , 1995, Int. J. Robotics Res..

[45]  José Luis Pons Rovira,et al.  A Closed-Loop Brain–Computer Interface Triggering an Active Ankle–Foot Orthosis for Inducing Cortical Neural Plasticity , 2014, IEEE Transactions on Biomedical Engineering.

[46]  Carlos Balaguer Nowadays Trends in Robotics and Automation in Construction Industry: Transition from Hard to Soft Robotics , 2004 .

[47]  Chang-soo Han "Human-Robot Cooperation Technology" An Ideal Midway Solution Heading Toward the Future of Robotics and Automation in Construction , 2011 .

[48]  Alexandre M. Amory,et al.  A Survey on Unmanned Surface Vehicles for Disaster Robotics: Main Challenges and Directions , 2019, Sensors.

[49]  Houtan Jebelli,et al.  Mobile EEG-Based Workers’ Stress Recognition by Applying Deep Neural Network , 2018, Advances in Informatics and Computing in Civil and Construction Engineering.

[50]  Klaus-Robert Müller,et al.  ℓ1-penalized linear mixed-effects models for high dimensional data with application to BCI , 2011, NeuroImage.

[51]  Stefan Haufe,et al.  Parieto-occipital alpha power indexes distraction during simulated car driving , 2008 .

[52]  Charles R. Crowell,et al.  Robot social presence and gender: Do females view robots differently than males? , 2008, 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[53]  Houtan Jebelli,et al.  EEG Signal-Processing Framework to Obtain High-Quality Brain Waves from an Off-the-Shelf Wearable EEG Device , 2018, J. Comput. Civ. Eng..

[54]  Iina Aaltonen,et al.  Refining levels of collaboration to support the design and evaluation of human-robot interaction in the manufacturing industry , 2018 .

[55]  Ellen S. Wilschut,et al.  Brain–machine interfaces in space: Using spontaneous rather than intentionally generated brain signals , 2010 .

[56]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update , 2018, Journal of neural engineering.

[57]  Christoph Richter,et al.  Human-Robot-Interaction for mobile industrial robot teams , 2019, Procedia CIRP.

[58]  Djamel Bouchaffra,et al.  Random Forest and Filter Bank Common Spatial Patterns for EEG-Based Motor Imagery Classification , 2014, 2014 5th International Conference on Intelligent Systems, Modelling and Simulation.

[59]  Liqing Zhang,et al.  EEG-based asynchronous BCI control of a car in 3D virtual reality environments , 2009 .

[60]  Helga Silaghi,et al.  Voice command of an industrial robot in a noisy environment , 2014, 2014 International Symposium on Fundamentals of Electrical Engineering (ISFEE).

[61]  Xinjun Sheng,et al.  Shared control of a robotic arm using non-invasive brain-computer interface and computer vision guidance , 2019, Robotics Auton. Syst..

[62]  Hang Zhang,et al.  Ubiquitous Log Odds: A Common Representation of Probability and Frequency Distortion in Perception, Action, and Cognition , 2012, Front. Neurosci..

[63]  Bin He,et al.  Brain–Computer Interfaces Using Sensorimotor Rhythms: Current State and Future Perspectives , 2014, IEEE Transactions on Biomedical Engineering.

[64]  Francesco Leali,et al.  Survey on human–robot collaboration in industrial settings: Safety, intuitive interfaces and applications , 2018, Mechatronics.

[65]  Yongfeng Luo,et al.  Data Missing Mechanism and Missing Data Real-Time Processing Methods in the Construction Monitoring of Steel Structures , 2015 .

[66]  Max Da Silva Simoes,et al.  Collision avoidance, virtual guides and advanced supervisory control teleoperation techniques for high-tech construction: framework design , 2014 .

[67]  M Congedo,et al.  A review of classification algorithms for EEG-based brain–computer interfaces , 2007, Journal of neural engineering.

[68]  Houtan Jebelli,et al.  Neurophysiological testing for assessing construction workers' task performance at virtual height , 2020 .

[69]  Klaus-Robert Müller,et al.  Machine learning for real-time single-trial EEG-analysis: From brain–computer interfacing to mental state monitoring , 2008, Journal of Neuroscience Methods.

[70]  Julie Huschilt,et al.  The use of socially assistive robots for dementia care. , 2012, Journal of gerontological nursing.

[71]  Adam Rogowski,et al.  Industrially oriented voice control system , 2012 .

[72]  Aude Billard,et al.  Safety issues in human-robot interactions , 2013, 2013 IEEE International Conference on Robotics and Automation.

[73]  Boyd C. Paulson,et al.  Automation and robotics for construction , 1985 .

[74]  Nikolaos G. Tsagarakis,et al.  Adaptation of robot physical behaviour to human fatigue in human-robot co-manipulation , 2016, 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids).

[75]  G. Pfurtscheller,et al.  How many people are able to operate an EEG-based brain-computer interface (BCI)? , 2003, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[76]  Klaus-Robert Müller,et al.  Subject-independent mental state classification in single trials , 2009, Neural Networks.

[77]  G. Pfurtscheller,et al.  Optimal spatial filtering of single trial EEG during imagined hand movement. , 2000, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[78]  R. Singla,et al.  Fixation of Delayed Union or Non-Union Posterior Cruciate Ligament Avulsion Fractures , 2014, Journal of orthopaedic surgery.

[79]  Jongwon Kim,et al.  Excavator tele-operation system using a human arm , 2009 .

[80]  Petr Novák,et al.  HUMAN – ROBOT COLLABORATION IN INDUSTRY , 2016 .

[81]  Cuntai Guan,et al.  EEG Data Space Adaptation to Reduce Intersession Nonstationarity in Brain-Computer Interface , 2013, Neural Computation.

[82]  Preeti Rani,et al.  Coal Mine Rescue Robot Simulation Using V-Rep and Python , 2019 .

[83]  B. Casanova,et al.  Disturbed Glucose Metabolism in Rat Neurons Exposed to Cerebrospinal Fluid Obtained from Multiple Sclerosis Subjects , 2017, Brain sciences.

[84]  Kazuhiro Kosuge,et al.  Progress and prospects of the human–robot collaboration , 2017, Autonomous Robots.

[85]  Carol C. Menassa,et al.  Task Allocation and Route Planning for Robotic Service Networks in Indoor Building Environments , 2017, J. Comput. Civ. Eng..

[86]  Bui Ngoc Minh Truong,et al.  Force reflecting joystick control for applications to bilateral teleoperation in construction machinery , 2017 .