Electronic skins and machine learning for intelligent soft robots

Developments in e-skins and machine learning may achieve tactile sensing and proprioception for autonomous, deployable soft robots. Soft robots have garnered interest for real-world applications because of their intrinsic safety embedded at the material level. These robots use deformable materials capable of shape and behavioral changes and allow conformable physical contact for manipulation. Yet, with the introduction of soft and stretchable materials to robotic systems comes a myriad of challenges for sensor integration, including multimodal sensing capable of stretching, embedment of high-resolution but large-area sensor arrays, and sensor fusion with an increasing volume of data. This Review explores the emerging confluence of e-skins and machine learning, with a focus on how roboticists can combine recent developments from the two fields to build autonomous, deployable soft robots, integrated with capabilities for informative touch and proprioception to stand up to the challenges of real-world environments.

[1]  R. Johansson,et al.  Tactile sensibility in the human hand: relative and absolute densities of four types of mechanoreceptive units in glabrous skin. , 1979, The Journal of physiology.

[2]  Allister F. McGuire,et al.  A skin-inspired organic digital mechanoreceptor , 2015, Science.

[3]  Fumiya Iida,et al.  Non-Destructive Robotic Assessment of Mango Ripeness via Multi-Point Soft Haptics , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[4]  Jakub W. Pachocki,et al.  Learning dexterous in-hand manipulation , 2018, Int. J. Robotics Res..

[5]  Jamie Paik,et al.  Closed-Loop Haptic Feedback Control Using a Self-Sensing Soft Pneumatic Actuator Skin. , 2020, Soft robotics.

[6]  Patricia A. Xu,et al.  Optical lace for synthetic afferent neural networks , 2019, Science Robotics.

[7]  R. Adam Bilodeau,et al.  Monolithic fabrication of sensors and actuators in a soft robotic gripper , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[8]  Pierre-Yves Oudeyer,et al.  Information-seeking, curiosity, and attention: computational and neural mechanisms , 2013, Trends in Cognitive Sciences.

[9]  Sheng Xu,et al.  Three-dimensional integrated stretchable electronics , 2018, Nature Electronics.

[10]  Yong-Lae Park,et al.  Sensorized, Flat, Pneumatic Artificial Muscle Embedded with Biomimetic Microfluidic Sensors for Proprioceptive Feedback. , 2019, Soft robotics.

[11]  Stefan Schaal,et al.  Understanding haptics by evolving mechatronic systems. , 2011, Progress in brain research.

[12]  Yasuo Kuniyoshi,et al.  Humanoid robot which can lift a 30kg box by whole body contact and tactile feedback , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[13]  Ross A. Knepper,et al.  A Deformable Interface for Human Touch Recognition Using Stretchable Carbon Nanotube Dielectric Elastomer Sensors and Deep Neural Networks. , 2017, Soft robotics.

[14]  Robert J. Wood,et al.  Soft curvature sensors for joint angle proprioception , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  Howie Choset,et al.  Continuum Robots for Medical Applications: A Survey , 2015, IEEE Transactions on Robotics.

[16]  Vincent Hayward,et al.  Large-Area Soft e-Skin: The Challenges Beyond Sensor Designs , 2019, Proceedings of the IEEE.

[17]  Hod Lipson,et al.  Unshackling evolution: evolving soft robots with multiple materials and a powerful generative encoding , 2013, GECCO '13.

[18]  Daniela Rus,et al.  Exploration of underwater life with an acoustically controlled soft robotic fish , 2018, Science Robotics.

[19]  Cecilia Laschi,et al.  Soft robot perception using embedded soft sensors and recurrent neural networks , 2019, Science Robotics.

[20]  Nawid Jamali,et al.  Material classification by tactile sensing using surface textures , 2010, 2010 IEEE International Conference on Robotics and Automation.

[21]  J. Randall Flanagan,et al.  Coding and use of tactile signals from the fingertips in object manipulation tasks , 2009, Nature Reviews Neuroscience.

[22]  Pascal Vincent,et al.  Representation Learning: A Review and New Perspectives , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Razvan Pascanu,et al.  Learning to Navigate in Complex Environments , 2016, ICLR.

[24]  Sergey Levine,et al.  Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations , 2017, Robotics: Science and Systems.

[25]  Kit-Hang Lee,et al.  Real-Time Surface Shape Sensing for Soft and Flexible Structures Using Fiber Bragg Gratings , 2019, IEEE Robotics and Automation Letters.

[26]  Tao Chen,et al.  Learning Exploration Policies for Navigation , 2019, ICLR.

[27]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[28]  R. Klatzky,et al.  Hand movements: A window into haptic object recognition , 1987, Cognitive Psychology.

[29]  Jivko Sinapov,et al.  Vibrotactile Recognition and Categorization of Surfaces by a Humanoid Robot , 2011, IEEE Transactions on Robotics.

[30]  Gordon Cheng,et al.  New materials and advances in making electronic skin for interactive robots , 2015, Adv. Robotics.

[31]  Heinrich M. Jaeger,et al.  JSEL: Jamming Skin Enabled Locomotion , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[32]  Andrew G. Gillies,et al.  Nanowire active-matrix circuitry for low-voltage macroscale artificial skin. , 2010, Nature materials.

[33]  M. J. Wells.  Tactile Discrimination of Surface Curvature and Shape by the Octopus , 1964 .

[34]  Gordon Cheng,et al.  Tactile-based active object discrimination and target object search in an unknown workspace , 2018, Autonomous Robots.

[35]  Filip Ilievski,et al.  Multigait soft robot , 2011, Proceedings of the National Academy of Sciences.

[36]  Zhenan Bao,et al.  A bioinspired flexible organic artificial afferent nerve , 2018, Science.

[37]  Jamie Kyujin Paik,et al.  An any-resolution pressure localization scheme using a soft capacitive sensor skin , 2018, 2018 IEEE International Conference on Soft Robotics (RoboSoft).

[38]  Zhenan Bao,et al.  Pursuing prosthetic electronic skin. , 2016, Nature materials.

[39]  Daniel M. Vogt,et al.  Embedded 3D Printing of Strain Sensors within Highly Stretchable Elastomers , 2014, Advanced materials.

[40]  Michelle C. Yuen,et al.  Morphing Robots Using Robotic Skins That Sculpt Clay , 2019, IEEE Robotics and Automation Letters.

[41]  Davide Scaramuzza,et al.  Dynamic obstacle avoidance for quadrotors with event cameras , 2020, Science Robotics.

[42]  Carmel Majidi,et al.  Machine Learning for Soft Robotic Sensing and Control , 2020, Adv. Intell. Syst..

[43]  Matthias Scheutz,et al.  The Tactile Ethics of Soft Robotics: Designing Wisely for Human-Robot Interaction. , 2017, Soft robotics.

[44]  Modris Greitans,et al.  Acceleration and Magnetic Sensor Network for Shape Sensing , 2016, IEEE Sensors Journal.

[45]  Gian Domenico Iannetti,et al.  Whole-Body Mapping of Spatial Acuity for Pain and Touch , 2014, Annals of neurology.

[46]  Michelle C. Yuen,et al.  OmniSkins: Robotic skins that turn inanimate objects into multifunctional robots , 2018, Science Robotics.

[47]  Nan Sun,et al.  Modular and Reconfigurable Wireless E‐Tattoos for Personalized Sensing , 2019, Advanced Materials Technologies.

[48]  Fulvio Mastrogiovanni,et al.  Skin spatial calibration using force/torque measurements , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[49]  Gordon Cheng,et al.  3D surface reconstruction for robotic body parts with artificial skins , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[50]  John Baldacchino,et al.  Learning to Be , 2014 .

[51]  Yong-Lae Park,et al.  Design and Fabrication of Soft Artificial Skin Using Embedded Microchannels and Liquid Conductors , 2012, IEEE Sensors Journal.

[52]  Matei Ciocarlie,et al.  Data-Driven Super-Resolution on a Tactile Dome , 2018, IEEE Robotics and Automation Letters.

[53]  Shigeki Sugano,et al.  Whole-body covering tactile interface for human robot coordination , 2002, Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No.02CH37292).

[54]  Mari Velonaki,et al.  Robotics and Autonomous Systems , 2014 .

[55]  Jan Peters,et al.  Stable reinforcement learning with autoencoders for tactile and visual data , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[56]  Helge J. Ritter,et al.  A Control Framework for Tactile Servoing , 2013, Robotics: Science and Systems.

[57]  Gordon Cheng,et al.  A Comprehensive Realization of Robot Skin: Sensors, Sensing, Control, and Applications , 2019, Proceedings of the IEEE.

[58]  Dmitry Berenson,et al.  Improving Soft Pneumatic Actuator fingers through integration of soft sensors, position and force control, and rigid fingernails , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[59]  ScheutzMatthias,et al.  The Tactile Ethics of Soft Robotics: Designing Wisely for Human–Robot Interaction , 2017 .

[60]  Oliver Brock,et al.  A novel type of compliant and underactuated robotic hand for dexterous grasping , 2016, Int. J. Robotics Res..

[61]  Veronica J. Santos,et al.  Biomimetic Tactile Sensor Array , 2008, Adv. Robotics.

[62]  Oussama Khatib,et al.  A hierarchically patterned, bioinspired e-skin able to detect the direction of applied pressure for robotics , 2018, Science Robotics.

[63]  Josep Amat,et al.  Human-Robot Interaction Based on a Sensitive Bumper Skin , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[64]  Michael T. Tolley,et al.  Design Considerations for 3D Printed, Soft, Multimaterial Resistive Sensors for Soft Robotics , 2019, Front. Robot. AI.

[65]  Joran W. Booth,et al.  Reconfigurable soft body trajectories using unidirectionally stretchable composite laminae , 2019, Nature Communications.

[66]  T. Someya,et al.  Conformable, flexible, large-area networks of pressure and thermal sensors with organic transistor active matrixes. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[67]  J. C. Stevens.  Aging and spatial acuity of touch. , 1992, Journal of gerontology.

[68]  Oliver Kroemer,et al.  Soft Magnetic Skin for Continuous Deformation Sensing , 2019, Adv. Intell. Syst..

[69]  Yonggang Huang,et al.  Stretchable and Foldable Silicon Integrated Circuits , 2008, Science.

[70]  Katsu Yamane,et al.  Design of a soft upper body robot for physical human-robot interaction , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[71]  Giulio Sandini,et al.  Tactile Sensing—From Humans to Humanoids , 2010, IEEE Transactions on Robotics.

[72]  Jaeha Kim,et al.  Electronic skins for soft, compact, reversible assembly of wirelessly activated fully soft robots , 2018, Science Robotics.

[73]  Jan Peters,et al.  Learning robot in-hand manipulation with tactile features , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[74]  Abhinav Gupta,et al.  The Curious Robot: Learning Visual Representations via Physical Interactions , 2016, ECCV.

[75]  Nawid Jamali,et al.  Majority Voting: Material Classification by Tactile Sensing Using Surface Texture , 2011, IEEE Transactions on Robotics.

[76]  Allison M. Okamura,et al.  A soft robot that navigates its environment through growth , 2017, Science Robotics.

[77]  Gordon Cheng,et al.  Realizing whole-body tactile interactions with a self-organizing, multi-modal artificial skin on a humanoid robot , 2015, Adv. Robotics.

[78]  Massimo Totaro,et al.  Toward Perceptive Soft Robots: Progress and Challenges , 2018, Advanced science.

[79]  D. Rus,et al.  Design, fabrication and control of soft robots , 2015, Nature.

[80]  Brian Litt,et al.  Flexible, Foldable, Actively Multiplexed, High-Density Electrode Array for Mapping Brain Activity in vivo , 2011, Nature Neuroscience.

[81]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[82]  Oussama Khatib,et al.  Ocean One: A Robotic Avatar for Oceanic Discovery , 2016, IEEE Robotics & Automation Magazine.

[83]  Si Li,et al.  A neuro-inspired artificial peripheral nervous system for scalable electronic skins , 2019, Science Robotics.

[84]  Tomonori Yamamoto,et al.  Use of tactile feedback to control exploratory movements to characterize object compliance , 2012, Front. Neurorobot..

[85]  Yonggang Huang,et al.  Two-dimensional materials in functional three-dimensional architectures with applications in photodetection and imaging , 2018, Nature Communications.

[86]  R. Johansson,et al.  Properties of cutaneous mechanoreceptors in the human hand related to touch sensation. , 1984, Human neurobiology.

[87]  Daniela Rus,et al.  Automated Recycling Separation Enabled by Soft Robotic Material Classification , 2019, 2019 2nd IEEE International Conference on Soft Robotics (RoboSoft).

[88]  Jan Peters,et al.  Learning inverse dynamics models with contacts , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[89]  Xiaochen Ren,et al.  A Low‐Operating‐Power and Flexible Active‐Matrix Organic‐Transistor Temperature‐Sensor Array , 2016, Advanced materials.

[90]  R. F. Shepherd,et al.  Soft optoelectronic sensory foams with proprioception , 2018, Science Robotics.

[91]  Jonathan Rossiter,et al.  Bodily Aware Soft Robots: Integration of Proprioceptive and Exteroceptive Sensors , 2017, ICRA.

[92]  Mark R. Cutkosky,et al.  Force and Tactile Sensors , 2008, Springer Handbook of Robotics.

[93]  David Kim,et al.  FlexSense: a transparent self-sensing deformable surface , 2014, UIST.

[94]  Gordon Cheng,et al.  Whole-Body Active Compliance Control for Humanoid Robots with Robot Skin , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[95]  Yong-Lae Park,et al.  Optically Sensorized Elastomer Air Chamber for Proprioceptive Sensing of Soft Pneumatic Actuators , 2020, IEEE Robotics and Automation Letters.

[96]  Mahmood Karimi,et al.  3D printed soft actuators for a legged robot capable of navigating unstructured terrain , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[97]  H. Shinoda,et al.  3D shape measuring sheet utilizing gravitational and geomagnetic fields , 2008, 2008 SICE Annual Conference.

[98]  M. Kaltenbrunner,et al.  An ultra-lightweight design for imperceptible plastic electronics , 2013, Nature.

[99]  I. Cohen,et al.  Stretchable surfaces with programmable 3D texture morphing for synthetic camouflaging skins , 2017, Science.

[100]  Mehmet Remzi Dogar,et al.  Haptic identification of objects using a modular soft robotic gripper , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[101]  Sergey Levine,et al.  Path integral guided policy search , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[102]  Gordon Cheng,et al.  3D spatial self-organization of a modular artificial skin , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[103]  Kevin O'Brien,et al.  Optoelectronically innervated soft prosthetic hand via stretchable optical waveguides , 2016, Science Robotics.

[104]  Wojciech Matusik,et al.  Learning the signatures of the human grasp using a scalable tactile glove , 2019, Nature.

[105]  Robert J. Wood,et al.  An integrated design and fabrication strategy for entirely soft, autonomous robots , 2016, Nature.

[106]  Máximo A. Roa,et al.  Planning in-hand object manipulation with multifingered hands considering task constraints , 2013, 2013 IEEE International Conference on Robotics and Automation.

[107]  Michael T. Tolley,et al.  Morphing Structure for Changing Hydrodynamic Characteristics of a Soft Underwater Walking Robot , 2019, IEEE Robotics and Automation Letters.

[108]  Jan Peters,et al.  Grip Stabilization of Novel Objects Using Slip Prediction , 2018, IEEE Transactions on Haptics.

[109]  Matteo Campanella,et al.  Detection of incipient object slippage by skin-like sensing and neural network processing , 1998, IEEE Trans. Syst. Man Cybern. Part B.

[110]  Rebecca K. Kramer,et al.  Sensor Skins: An Overview , 2016 .

[111]  Sigurd Wagner,et al.  Electronic Skin: Architecture and Components , 2004 .

[112]  I. Park,et al.  Stretchable, Skin‐Mountable, and Wearable Strain Sensors and Their Potential Applications: A Review , 2016 .

[113]  Katsu Yamane,et al.  Design and fabrication of a soft robotic hand and arm system , 2018, 2018 IEEE International Conference on Soft Robotics (RoboSoft).

[114]  Benjamin C. K. Tee,et al.  Flexible polymer transistors with high pressure sensitivity for application in electronic skin and health monitoring , 2013, Nature Communications.

[115]  Boris Murmann,et al.  Skin electronics from scalable fabrication of an intrinsically stretchable transistor array , 2018, Nature.

[116]  Gerald E. Loeb,et al.  Bayesian Exploration for Intelligent Identification of Textures , 2012, Front. Neurorobot..

[117]  D. Dewsbury,et al.  Octopus: Physiology and behaviour of an advanced invertebrate. , 1978 .

[118]  Cecilia Laschi,et al.  Control Strategies for Soft Robotic Manipulators: A Survey. , 2018, Soft robotics.

[119]  Robert J. Wood,et al.  Wearable soft sensing suit for human gait measurement , 2014, Int. J. Robotics Res..

[120]  Kaspar Althoefer,et al.  Tactile sensing for dexterous in-hand manipulation in robotics-A review , 2011 .

[121]  Sungho Jo,et al.  Deep Full-Body Motion Network for a Soft Wearable Motion Sensing Suit , 2019, IEEE/ASME Transactions on Mechatronics.

[122]  Carmel Majidi,et al.  A biosensing soft robot: Autonomous parsing of chemical signals through integrated organic and inorganic interfaces , 2019, Science Robotics.

[123]  T. Someya,et al.  A Rubberlike Stretchable Active Matrix Using Elastic Conductors , 2008, Science.

[124]  Sergey Levine,et al.  One-shot learning of manipulation skills with online dynamics adaptation and neural network priors , 2015, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[125]  Junghan Kwon,et al.  Adaptive Calibration of Soft Sensors Using Optimal Transportation Transfer Learning for Mass Production and Long‐Term Usage , 2020, Adv. Intell. Syst..

[126]  Nikolaus Correll,et al.  A soft pneumatic actuator that can sense grasp and touch , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[127]  Sami Haddadin,et al.  Physical Human-Robot Interaction , 2016, Springer Handbook of Robotics, 2nd Ed..

[128]  Henrik I. Christensen,et al.  Custom soft robotic gripper sensor skins for haptic object visualization , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[129]  Danica Kragic,et al.  Learning of grasp adaptation through experience and tactile sensing , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[130]  Jürgen Schmidhuber,et al.  Learning tactile skills through curious exploration , 2012, Front. Neurorobot..