A Survey of Teleceptive Sensing for Wearable Assistive Robotic Devices

Teleception is defined as sensing that occurs remotely, with no physical contact with the object being sensed. To emulate innate control systems of the human body, a control system for a semi- or fully autonomous assistive device not only requires feedforward models of desired movement, but also the environmental or contextual awareness that could be provided by teleception. Several recent publications present teleception modalities integrated into control systems and provide preliminary results, for example, for performing hand grasp prediction or endpoint control of an arm assistive device; and gait segmentation, forward prediction of desired locomotion mode, and activity-specific control of a prosthetic leg or exoskeleton. Collectively, several different approaches to incorporating teleception have been used, including sensor fusion, geometric segmentation, and machine learning. In this paper, we summarize the recent and ongoing published work in this promising new area of research.

[1]  Levi J. Hargrove,et al.  A Novel Method for Bilateral Gait Segmentation Using a Single Thigh-Mounted Depth Sensor and IMU , 2018, 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob).

[2]  Tao Liu,et al.  Gait Analysis Using Wearable Sensors , 2012, Sensors.

[3]  Fan Zhang,et al.  Preliminary design of a terrain recognition system , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[4]  Lee E. Weiss,et al.  Dynamic sensor-based control of robots with visual feedback , 1987, IEEE Journal on Robotics and Automation.

[5]  Jaydev P. Desai,et al.  Towards a Robotic Hand Rehabilitation Exoskeleton for Stroke Therapy , 2014 .

[6]  José del R. Millán,et al.  Brain-Controlled Wheelchairs: A Robotic Architecture , 2013, IEEE Robotics & Automation Magazine.

[7]  Wen Gao,et al.  Illumination normalization for robust face recognition against varying lighting conditions , 2003, 2003 IEEE International SOI Conference. Proceedings (Cat. No.03CH37443).

[8]  Strahinja Došen,et al.  Transradial prosthesis: artificial vision for control of prehension. , 2011, Artificial organs.

[9]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[10]  João Yoshiyuki Ishihara,et al.  Estimation of foot orientation with respect to ground for an above knee robotic prosthesis , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[11]  Gerald C. Holst,et al.  CMOS/CCD sensors and camera systems , 2007 .

[12]  Tobi Delbrück,et al.  A 128$\times$ 128 120 dB 15 $\mu$s Latency Asynchronous Temporal Contrast Vision Sensor , 2008, IEEE Journal of Solid-State Circuits.

[13]  Graham Morgan,et al.  Deep learning-based artificial vision for grasp classification in myoelectric hands , 2017, Journal of neural engineering.

[14]  Dr.-Ing. Thorsten Ringbeck A 3 D TIME OF FLIGHT CAMERA FOR OBJECT DETECTION , 2007 .

[15]  Jonathon W. Sensinger,et al.  Virtual Constraint Control of a Powered Prosthetic Leg: From Simulation to Experiments With Transfemoral Amputees , 2014, IEEE Transactions on Robotics.

[16]  Brenna Argall,et al.  Automated perception of safe docking locations with alignment information for assistive wheelchairs , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[17]  John T. Wen,et al.  A Sensor-Based Dual-Arm Tele-Robotic System , 2015, IEEE Transactions on Automation Science and Engineering.

[18]  Peter J Beek,et al.  Online gait event detection using a large force platform embedded in a treadmill. , 2008, Journal of biomechanics.

[19]  Timothy Bretl,et al.  Automatic grasp selection using a camera in a hand prosthesis , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[20]  Nitish V. Thakor,et al.  Demonstration of a Semi-Autonomous Hybrid Brain–Machine Interface Using Human Intracranial EEG, Eye Tracking, and Computer Vision to Control a Robotic Upper Limb Prosthetic , 2014, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[21]  Michael Goldfarb,et al.  Design and Control of a Powered Transfemoral Prosthesis , 2008, Int. J. Robotics Res..

[22]  Wen Zhang,et al.  Environmental Features Recognition for Lower Limb Prostheses Toward Predictive Walking , 2019, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[23]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[24]  L. M. Peterson,et al.  Laser radar in robotics , 1996 .

[25]  Francisco Javier Ferrández Pastor,et al.  A vision based proposal for classification of normal and abnormal gait using RGB camera , 2016, J. Biomed. Informatics.

[26]  Avinash C. Kak,et al.  Vision for Mobile Robot Navigation: A Survey , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[27]  Amnon Shashua,et al.  The Quotient Image: Class-Based Re-Rendering and Recognition with Varying Illuminations , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Michael Goldfarb,et al.  Multiclass Real-Time Intent Recognition of a Powered Lower Limb Prosthesis , 2010, IEEE Transactions on Biomedical Engineering.

[29]  R. Stein Peripheral control of movement. , 1974, Physiological reviews.

[30]  Strahinja Došen,et al.  Cognitive vision system for control of dexterous prosthetic hands: Experimental evaluation , 2010, Journal of NeuroEngineering and Rehabilitation.

[31]  Cristina P. Santos,et al.  Environment-Aware Locomotion Mode Transition Prediction System , 2019, 2019 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC).

[32]  Elaine Biddiss,et al.  Consumer design priorities for upper limb prosthetics , 2007, Disability and rehabilitation. Assistive technology.

[33]  M. A. MacIver,et al.  Sensory acquisition in active sensing systems , 2006, Journal of Comparative Physiology A.

[34]  Silvestro Micera,et al.  Intent Prediction Based on Biomechanical Coordination of EMG and Vision-Filtered Gaze for End-Point Control of an Arm Prosthesis , 2020, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[35]  Konrad P. Kording,et al.  Real-Time Evaluation of a Noninvasive Neuroprosthetic Interface for Control of Reach , 2013, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[36]  T. Delbruck,et al.  > Replace This Line with Your Paper Identification Number (double-click Here to Edit) < 1 , 2022 .

[37]  Dario J. Villarreal,et al.  Obstacle Recognition using Computer Vision and Convolutional Neural Networks for Powered Prosthetic Leg Applications , 2019, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[38]  Robert Kelly,et al.  A novel approach of prosthetic arm control using computer vision, biosignals, and motion capture , 2014, 2014 IEEE Symposium on Computational Intelligence in Robotic Rehabilitation and Assistive Technologies (CIR2AT).

[39]  Levi J Hargrove,et al.  Subject- and Environment-Based Sensor Variability for Wearable Lower-Limb Assistive Devices , 2019, Sensors.

[40]  David J. Kriegman,et al.  What Is the Set of Images of an Object Under All Possible Illumination Conditions? , 1998, International Journal of Computer Vision.

[41]  Sebastian Thrun,et al.  3D shape scanning with a time-of-flight camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[42]  Jianhua Wang,et al.  Real-Time Stairs Geometric Parameters Estimation for Lower Limb Rehabilitation Exoskeleton , 2018, 2018 Chinese Control And Decision Conference (CCDC).

[43]  Nitish V. Thakor,et al.  Orientation estimation and grasp type detection of household objects for upper limb prostheses with dynamic vision sensor , 2016, 2016 IEEE Biomedical Circuits and Systems Conference (BioCAS).

[44]  Jing Xu,et al.  Vision-Assisted Autonomous Lower-Limb Exoskeleton Robot , 2019, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[45]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[46]  R. Simpson Smart wheelchairs: A literature review. , 2005, Journal of rehabilitation research and development.

[47]  Huseyin Atakan Varol,et al.  Human grasping database for activities of daily living with depth, color and kinematic data streams , 2018, Scientific Data.

[48]  Dario Farina,et al.  Stereovision and augmented reality for closed-loop control of grasping in hand prostheses , 2014, Journal of neural engineering.

[49]  Barbara Caputo,et al.  Megane Pro: Myo-electricity, visual and gaze tracking data acquisitions to improve hand prosthetics , 2017, 2017 International Conference on Rehabilitation Robotics (ICORR).

[50]  Alexander Verl,et al.  A Radar-Based Terrain Mapping Approach for Stair Detection Towards Enhanced Prosthetic Foot Control , 2018, 2018 7th IEEE International Conference on Biomedical Robotics and Biomechatronics (Biorob).

[51]  Ronen Basri,et al.  Lambertian reflectance and linear subspaces , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[52]  Huseyin Atakan Varol,et al.  A feasibility study of depth image based intent recognition for lower limb prostheses , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[53]  Peter C. Cheeseman,et al.  Estimating uncertain spatial relationships in robotics , 1986, Proceedings. 1987 IEEE International Conference on Robotics and Automation.

[54]  Youngjoon Chee,et al.  A Virtual Blind Cane Using a Line Laser-Based Vision System and an Inertial Measurement Unit , 2016, Sensors.

[55]  Viii Supervisor Sonar-Based Real-World Mapping and Navigation , 2001 .

[56]  大卫·S·霍尔 High definition lidar system , 2007 .

[57]  Levi J. Hargrove,et al.  Depth Sensing for Improved Control of Lower Limb Prostheses , 2015, IEEE Transactions on Biomedical Engineering.

[58]  Hiroshi Okumura,et al.  A semi-automatic control method for myoelectric prosthetic hand based on image information of objects , 2017, 2017 International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS).

[59]  Ashutosh Saxena,et al.  Learning Depth from Single Monocular Images , 2005, NIPS.

[60]  Levi J. Hargrove,et al.  Recognition of ascending stairs from 2D images for control of powered lower limb prostheses , 2015, 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER).

[61]  R. Lange,et al.  Solid-state time-of-flight range camera , 2001 .

[62]  Chenglong Fu,et al.  Directional PointNet: 3D Environmental Classification for Wearable Robotics , 2019, ArXiv.

[63]  Beiwen Li,et al.  Structured Light Techniques and Applications , 2016 .

[64]  Alexander Wong,et al.  Preliminary Design of an Environment Recognition System for Controlling Robotic Lower-Limb Prostheses and Exoskeletons , 2019, 2019 IEEE 16th International Conference on Rehabilitation Robotics (ICORR).

[65]  Eric Monacelli,et al.  Intelligent Assistive Exoskeleton with Vision Based Interface , 2008, ICOST.

[66]  Sebastian Thrun,et al.  Towards fully autonomous driving: Systems and algorithms , 2011, 2011 IEEE Intelligent Vehicles Symposium (IV).

[67]  Ciara M O'Connor,et al.  Automatic detection of gait events using kinematic data. , 2007, Gait & posture.

[68]  Robert Riener,et al.  A survey of sensor fusion methods in wearable robotics , 2015, Robotics Auton. Syst..

[69]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[70]  David Lee,et al.  Dual Aperture Photography: Image and Depth from a Mobile Camera , 2015, 2015 IEEE International Conference on Computational Photography (ICCP).

[71]  John Krumm,et al.  Location-aware computing comes of age , 2004, Computer.

[72]  T. Ringbeck,et al.  © Author(s) 2007. This work is licensed under a Creative Commons License. Multidimensional measurement by using 3-D PMD sensors , 2022 .

[73]  Dgk Madusanka Development of a vision aided reach-to-grasp path planning and controlling method for trans-humeral robotic prostheses , 2018 .

[74]  Ming Liu,et al.  Effects of Locomotion Mode Recognition Errors on Volitional Control of Powered Above-Knee Prostheses , 2015, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[75]  Tingfang Yan,et al.  A Locomotion Recognition System Using Depth Images , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[76]  Otmar Hilliges,et al.  Video-based Prediction of Hand-grasp Preshaping with Application to Prosthesis Control , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[77]  Gholamreza Khademi,et al.  Convolutional Neural Networks for Environmentally Aware Locomotion Mode Recognition of Lower-Limb Amputees , 2019 .

[78]  Kari Pulli,et al.  Real-time computer vision with OpenCV , 2012, Commun. ACM.

[79]  Francois Blais,et al.  Practical Considerations For A Design Of A High Precision 3-D Laser Scanner System , 1988, Other Conferences.

[80]  Etienne Burdet,et al.  An unobtrusive vision system to reduce the cognitive burden of hand prosthesis control , 2014, 2014 13th International Conference on Control Automation Robotics & Vision (ICARCV).

[81]  Jonathon W. Sensinger,et al.  Preliminary evaluation of a new control approach to achieve speed adaptation in robotic transfemoral prostheses , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[82]  R. D Seidler,et al.  Feedforward and feedback processes in motor control , 2004, NeuroImage.

[83]  Huseyin Atakan Varol,et al.  User-Independent Intent Recognition for Lower Limb Prostheses Using Depth Sensing , 2018, IEEE Transactions on Biomedical Engineering.

[84]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[85]  P. Benner The Cambridge Handbook of Situated Cognition: Book Review , 2010 .

[86]  Shuihua Wang,et al.  Detecting stairs and pedestrian crosswalks for the blind by RGBD camera , 2012, 2012 IEEE International Conference on Bioinformatics and Biomedicine Workshops.

[87]  Ming Liu,et al.  Development of an Environment-Aware Locomotion Mode Recognition System for Powered Lower Limb Prostheses , 2016, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[88]  François Blais Review of 20 years of range sensor development , 2004, J. Electronic Imaging.

[89]  Dario Farina,et al.  Sensor fusion and computer vision for context-aware control of a multi degree-of-freedom prosthesis , 2015, Journal of neural engineering.

[90]  Natasha Merat,et al.  Control Task Substitution in Semiautomated Driving , 2012, Hum. Factors.

[91]  Peisen S. Huang,et al.  Novel method for structured light system calibration , 2006 .