Soft robot perception using embedded soft sensors and recurrent neural networks

Recurrent neural networks with an unstructured redundant soft sensor topology allow robust multimodal proprioceptive capabilities. Recent work has begun to explore the design of biologically inspired soft robots composed of soft, stretchable materials for applications including the handling of delicate materials and safe interaction with humans. However, the solid-state sensors traditionally used in robotics are unable to capture the high-dimensional deformations of soft systems. Embedded soft resistive sensors have the potential to address this challenge. However, both the soft sensors—and the encasing dynamical system—often exhibit nonlinear time-variant behavior, which makes them difficult to model. In addition, the problems of sensor design, placement, and fabrication require a great deal of human input and previous knowledge. Drawing inspiration from the human perceptive system, we created a synthetic analog. Our synthetic system builds models using a redundant and unstructured sensor topology embedded in a soft actuator, a vision-based motion capture system for ground truth, and a general machine learning approach. This allows us to model an unknown soft actuated system. We demonstrate that the proposed approach is able to model the kinematics of a soft continuum actuator in real time while being robust to sensor nonlinearities and drift. In addition, we show how the same system can estimate the applied forces while interacting with external objects. The role of action in perception is also presented. This approach enables the development of force and deformation models for soft robotic systems, which can be useful for a variety of applications, including human-robot interaction, soft orthotics, and wearable robotics.

[1]  Matteo Cianchetti,et al.  Soft robotics: Technologies and systems pushing the boundaries of robot abilities , 2016, Science Robotics.

[2]  R. F. Shepherd,et al.  Soft optoelectronic sensory foams with proprioception , 2018, Science Robotics.

[3]  Nikolaus Correll,et al.  Integrated proximity, contact and force sensing using elastomer-embedded commodity proximity sensors , 2018, Auton. Robots.

[4]  Veronica J. Santos,et al.  Biomimetic Tactile Sensor Array , 2008, Adv. Robotics.

[5]  Luheng Wang,et al.  Study on compressive resistance creep and recovery of flexible pressure sensitive material based on carbon black filled silicone rubber composite , 2011 .

[6]  S. Gandevia,et al.  The proprioceptive senses: their roles in signaling body shape, body position and movement, and muscle force. , 2012, Physiological reviews.

[7]  Daniel M. Vogt,et al.  Soft Somatosensitive Actuators via Embedded 3D Printing , 2018, Advanced materials.

[8]  Oliver Brock,et al.  A method for sensorizing soft actuators and its application to the RBO hand 2 , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[9]  Massimo Totaro,et al.  Toward Perceptive Soft Robots: Progress and Challenges , 2018, Advanced science.

[10]  D. Rus,et al.  Design, fabrication and control of soft robots , 2015, Nature.

[11]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[12]  K. Nagai,et al.  SCN output drives the autonomic nervous system: with special reference to the autonomic function related to the regulation of glucose metabolism. , 1996, Progress in brain research.

[13]  Daniel M. Vogt,et al.  Embedded 3D Printing of Strain Sensors within Highly Stretchable Elastomers , 2014, Advanced materials.

[14]  Michael Thomas Tolley,et al.  3D printed resistive soft sensors , 2018, 2018 IEEE International Conference on Soft Robotics (RoboSoft).

[15]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[16]  P. Matthews,et al.  Mammalian muscle receptors and their central actions , 1974 .

[17]  Anne C. Sittig,et al.  The precision of proprioceptive position sense , 1998, Experimental Brain Research.

[18]  Mehmet Remzi Dogar,et al.  Haptic identification of objects using a modular soft robotic gripper , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[19]  Rebecca K. Kramer,et al.  Hyperelastic pressure sensing with a liquid-embedded elastomer , 2010 .

[20]  Kevin O'Brien,et al.  Optoelectronically innervated soft prosthetic hand via stretchable optical waveguides , 2016, Science Robotics.

[21]  I. Park,et al.  Stretchable, Skin‐Mountable, and Wearable Strain Sensors and Their Potential Applications: A Review , 2016 .

[22]  F. Rösler,et al.  Plasticity of multisensory dorsal stream functions: evidence from congenitally blind and sighted adults. , 2010, Restorative neurology and neuroscience.

[23]  Michael Thomas Tolley,et al.  Differential pressure control of 3D printed soft fluidic actuators , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[24]  Filip Ilievski,et al.  Multigait soft robot , 2011, Proceedings of the National Academy of Sciences.

[25]  T J Allen,et al.  Limb position sense, proprioceptive drift and muscle thixotropy at the human elbow joint , 2014, The Journal of physiology.

[26]  Robert J. van Beers,et al.  How humans combine simultaneous proprioceptive and visual position information , 1996, Experimental Brain Research.

[27]  K. Hata,et al.  A stretchable carbon nanotube strain sensor for human-motion detection. , 2011, Nature nanotechnology.

[28]  Mats Jackson,et al.  Bending angle prediction and control of soft pneumatic actuators with embedded flex sensors: a data-driven approach , 2017 .

[29]  Sarthak Misra,et al.  Force sensing in continuum manipulators using fiber Bragg grating sensors , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[30]  Nantachai Sornkarn,et al.  Can a Soft Robotic Probe Use Stiffness Control Like a Human Finger to Improve Efficacy of Haptic Perception? , 2017, IEEE Transactions on Haptics.

[31]  Kai Xu,et al.  An Investigation of the Intrinsic Force Sensing Capabilities of Continuum Robots , 2008, IEEE Transactions on Robotics.

[32]  Cecilia Laschi,et al.  Control Strategies for Soft Robotic Manipulators: A Survey. , 2018, Soft robotics.

[33]  Robert J. Wood,et al.  Wearable soft sensing suit for human gait measurement , 2014, Int. J. Robotics Res..

[34]  Sungho Jo,et al.  Deep Full-Body Motion Network for a Soft Wearable Motion Sensing Suit , 2019, IEEE/ASME Transactions on Mechatronics.

[35]  G. Tröster,et al.  Sensor for Measuring Strain in Textile , 2008, Sensors.

[36]  Cagdas D. Onal,et al.  A precise embedded curvature sensor module for soft-bodied robots , 2015 .

[37]  Monica Gori,et al.  Auditory and proprioceptive spatial impairments in blind children and adults. , 2017, Developmental science.

[38]  Ronald J. Williams,et al.  A Learning Algorithm for Continually Running Fully Recurrent Neural Networks , 1989, Neural Computation.

[39]  Yihu Song,et al.  Time-dependent uniaxial piezoresistive behavior of high-density polyethylene/short carbon fiber conductive composites , 2004 .

[40]  Nabil Simaan,et al.  Finding lost wrenches: Using continuum robots for contact detection and estimation of contact location , 2010, 2010 IEEE International Conference on Robotics and Automation.

[41]  Henrik I. Christensen,et al.  Custom soft robotic gripper sensor skins for haptic object visualization , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[42]  Robert J. Webster,et al.  On the inseparable nature of sensor selection, sensor placement, and state estimation for continuum robots or “where to put your sensors and how to use them” , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[43]  Daniel M. Vogt,et al.  Design and Characterization of a Soft Multi-Axis Force Sensor Using Embedded Microfluidic Channels , 2013, IEEE Sensors Journal.

[44]  Sungho Jo,et al.  Use of Deep Learning for Characterization of Microfluidic Soft Sensors , 2018, IEEE Robotics and Automation Letters.

[45]  Wei Zhang,et al.  Simultaneous measurement of strain and curvature using superstructure fibre Bragg gratings , 2002 .

[46]  Ross A. Knepper,et al.  A Deformable Interface for Human Touch Recognition Using Stretchable Carbon Nanotube Dielectric Elastomer Sensors and Deep Neural Networks. , 2017, Soft robotics.

[47]  Nabil Simaan,et al.  Configuration and joint feedback for enhanced performance of multi-segment continuum robots , 2011, 2011 IEEE International Conference on Robotics and Automation.

[48]  Stephen A. Morin,et al.  Soft Robotics: Review of Fluid‐Driven Intrinsically Soft Devices; Manufacturing, Sensing, Control, and Applications in Human‐Robot Interaction   , 2017 .

[49]  Dmitry Berenson,et al.  Improving Soft Pneumatic Actuator fingers through integration of soft sensors, position and force control, and rigid fingernails , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).