Human-Machine Interaction through Advanced Haptic Sensors: A Piezoelectric Sensory Glove with Edge Machine Learning for Gesture and Object Recognition

Human–machine interaction (HMI) refers to systems enabling communication between machines and humans. Systems for human–machine interfaces have advanced significantly in terms of materials, device design, and production methods. Energy supply units, logic circuits, sensors, and data storage units must be flexible, stretchable, undetectable, biocompatible, and self-healing to act as human–machine interfaces. This paper discusses the technologies for providing different haptic feedback of different natures. Notably, the physiological mechanisms behind touch perception are reported, along with a classification of the main haptic interfaces. Afterward, a comprehensive overview of wearable haptic interfaces is presented, comparing them in terms of cost, the number of integrated actuators and sensors, their main haptic feedback typology, and their future application. Additionally, a review of sensing systems that use haptic feedback technologies—specifically, smart gloves—is given by going through their fundamental technological specifications and key design requirements. Furthermore, useful insights related to the design of the next-generation HMI devices are reported. Lastly, a novel smart glove based on thin and conformable AlN (aluminum nitride) piezoelectric sensors is demonstrated. Specifically, the device acquires and processes the signal from the piezo sensors to classify performed gestures through an onboard machine learning (ML) algorithm. Then, the design and testing of the electronic conditioning section of AlN-based sensors integrated into the smart glove are shown. Finally, the architecture of a wearable visual-tactile recognition system is presented, combining visual data acquired by a micro-camera mounted on the user’s glass with the haptic ones provided by the piezoelectric sensors.

[1]  J. A. Rogers,et al.  Materials and devices for immersive virtual reality , 2022, Nature Reviews Materials.

[2]  A. Al-hinnawi,et al.  An Energy-Autonomous Smart Shirt Employing Wearable Sensors for Users’ Safety and Protection in Hazardous Workplaces , 2022, Applied Sciences.

[3]  Oliver Ozioko,et al.  Smart Tactile Gloves for Haptic Interaction, Communication, and Rehabilitation , 2021, Adv. Intell. Syst..

[4]  Pan Hui,et al.  A Survey on Haptic Technologies for Mobile Augmented Reality , 2017, ACM Comput. Surv..

[5]  R. Fazio,et al.  A BLE-Connected Piezoresistive and Inertial Chest Band for Remote Monitoring of the Respiratory Activity by an Android Application: Hardware Design and Software Optimization , 2022, Future Internet.

[6]  A. Kulasekera,et al.  A Layer Jamming Soft Glove for Hand Tremor Suppression , 2021, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[7]  Yiming Liu,et al.  Recent advances in multi-mode haptic feedback technologies towards wearable interfaces , 2021, Materials Today Physics.

[8]  R. Dahiya,et al.  Disposable Electrochemical Sensor Using Graphene Oxide–Chitosan Modified Carbon-Based Electrodes for the Detection of Tyrosine , 2021, IEEE Sensors Journal.

[9]  M. Todaro,et al.  Conformable AlN Piezoelectric Sensors as a Non-invasive Approach for Swallowing Disorder Assessment , 2021, ACS sensors.

[10]  R. Dahiya,et al.  Textile-Based Stretchable Microstrip Antenna with Intrinsic Strain Sensing , 2021, ACS applied electronic materials.

[11]  M. Mariello,et al.  Conformal, Ultra‐thin Skin‐Contact‐Actuated Hybrid Piezo/Triboelectric Wearable Sensor Based on AlN and Parylene‐Encapsulated Elastomeric Blend , 2021, Advanced Functional Materials.

[12]  Kai Heinrich,et al.  Machine learning and deep learning , 2021, Electron. Mark..

[13]  Jonathan Rossiter,et al.  B:Ionic Glove: A Soft Smart Wearable Sensory Feedback Device for Upper Limb Robotic Prostheses , 2021, IEEE Robotics and Automation Letters.

[14]  Hanbit Jin,et al.  Recent Advances and Opportunities of Active Materials for Haptic Technologies in Virtual and Augmented Reality , 2021, Advanced Functional Materials.

[15]  Domingo Mery,et al.  An Efficient Point-Matching Method Based on Multiple Geometrical Hypotheses , 2021, Electronics.

[16]  Kian Meng Yap,et al.  Haptic Interfaces for Virtual Reality: Challenges and Research Directions , 2021, IEEE Access.

[17]  Erik G. Larsson,et al.  Towards 6G wireless communication networks: vision, enabling technologies, and new paradigm shifts , 2020, Science China Information Sciences.

[18]  Nicola Ivan Giannoccaro,et al.  Solar-Powered Deep Learning-Based Recognition System of Daily Used Objects and Human Faces for Assistance of the Visually Impaired , 2020, Energies.

[19]  Tianyiyi He,et al.  Technologies toward next generation human machine interfaces: From machine learning enhanced tactile sensing to neuromorphic sensory systems , 2020, Applied Physics Reviews.

[20]  Mario Milazzo,et al.  Haptic-based touch detection for collaborative robots in welding applications , 2020, Robotics Comput. Integr. Manuf..

[21]  H. Shea,et al.  Multimode Hydraulically Amplified Electrostatic Actuators for Wearable Haptics , 2020, Advanced materials.

[22]  R. Dahiya,et al.  A Wearable Supercapacitor Based on Conductive PEDOT:PSS‐Coated Cloth and a Sweat Electrolyte , 2020, Advanced materials.

[23]  Samuel E. Root,et al.  Electropneumotactile Stimulation: Multimodal Haptic Actuators Enabled by a Stretchable Conductive Polymer on Inflatable Pockets , 2020, Advanced materials technologies.

[24]  Qiongfeng Shi,et al.  Haptic-feedback smart glove as a creative human-machine interface (HMI) for virtual/augmented reality applications , 2020, Science Advances.

[25]  Roberto de Fazio,et al.  A Multi-Source Harvesting System Applied to Sensor-Based Smart Garments for Monitoring Workers’ Bio-Physical Parameters in Harsh Environments , 2020 .

[26]  Yan Gao,et al.  In Contact: Pinching, Squeezing and Twisting for Mediated Social Touch , 2020, CHI Extended Abstracts.

[27]  Seongho Kim,et al.  Two-Dimensional Thermal Haptic Module Based on a Flexible Thermoelectric Device. , 2020, Soft robotics.

[28]  Weiliang Xu,et al.  Co-Actuation: A Method for Achieving High Stiffness and Low Inertia for Haptic Devices , 2020, IEEE Transactions on Haptics.

[29]  Ahmed W. Shehata,et al.  Mechanotactile Sensory Feedback Improves Embodiment of a Prosthetic Hand During Active Use , 2020, Frontiers in Neuroscience.

[30]  Yasar Ayaz,et al.  Development and Testing of a Wearable Vibrotactile Haptic Feedback System for Proprioceptive Rehabilitation , 2020, IEEE Access.

[31]  Changhong Yang,et al.  Towards Multifunctional Electronics: Flexible NBT-Based Film with Large Electrocaloric Effect and High Energy Storage Property. , 2020, ACS applied materials & interfaces.

[32]  Dario Floreano,et al.  Stretchable pumps for soft machines , 2019, Nature.

[33]  M. Garrad,et al.  A soft matter computer for soft robots , 2019, Science Robotics.

[34]  Domenico Prattichizzo,et al.  Can Wearable Haptic Devices Foster the Embodiment of Virtual Limbs? , 2019, IEEE Transactions on Haptics.

[35]  Qiang He,et al.  Triboelectric vibration sensor for a human-machine interface built on ubiquitous surfaces , 2019, Nano Energy.

[36]  Felix G. Hamza-Lup,et al.  Haptic Systems in User Interfaces: State of the Art Survey , 2019, ACM Southeast Regional Conference.

[37]  Chuan Wang,et al.  Physical and Chemical Sensing With Electronic Skin , 2019, Proceedings of the IEEE.

[38]  Michael K. Southworth,et al.  Extended Reality in Medical Practice , 2019, Current Treatment Options in Cardiovascular Medicine.

[39]  Tae-Heon Yang,et al.  Design, modeling, and evaluation of a slim haptic actuator based on electrorheological fluid , 2019, Journal of Intelligent Material Systems and Structures.

[40]  Yiping Wang,et al.  Toward artificial intelligent self-cooling electronic skins: Large electrocaloric effect in all-inorganic flexible thin films at room temperature , 2019, Journal of Materiomics.

[41]  D. Diao,et al.  Intelligently detecting and identifying liquids leakage combining triboelectric nanogenerator based self-powered sensor with machine learning , 2019, Nano Energy.

[42]  Xun Liu,et al.  Haptic Codecs for the Tactile Internet , 2019, Proceedings of the IEEE.

[43]  Ravinder Dahiya,et al.  E-Skin: From Humanoids to Humans , 2019, Proc. IEEE.

[44]  K. Valaskova,et al.  The Power of Human–machine Collaboration: Artificial Intelligence, Business Automation, and the Smart Economy , 2018 .

[45]  Jaeyoung Park,et al.  Haptic Feedback to the Palm and Fingers for Improved Tactile Perception of Large Objects , 2018, UIST.

[46]  Zhong Lin Wang,et al.  Keystroke Dynamics Identification Based on Triboelectric Nanogenerator for Intelligent Keyboard Using Deep Learning Method , 2018, Advanced Materials Technologies.

[47]  Marilyn Baird,et al.  The implementation of a haptic feedback virtual reality simulation clinic with dynamic patient interaction and communication for medical imaging students , 2018, Journal of medical radiation sciences.

[48]  Karen B. Chen,et al.  Immersion of virtual reality for rehabilitation - Review. , 2018, Applied ergonomics.

[49]  Maria Chiara Carrozza,et al.  Haptic-assistive technologies for audition and vision sensory disabilities , 2018, Disability and rehabilitation. Assistive technology.

[50]  Timothy Bretl,et al.  Controlling sensation intensity for electrotactile stimulation in human-machine interfaces , 2018, Science Robotics.

[51]  Zhong Lin Wang,et al.  Keystroke dynamics enabled authentication and identification using triboelectric nanogenerator array , 2018 .

[52]  Chunfang Liu,et al.  3D human gesture capturing and recognition by the IMMU-based data glove , 2018, Neurocomputing.

[53]  Karl F. MacDorman,et al.  Review of constraints on vision-based gesture recognition for human-computer interaction , 2018, IET Comput. Vis..

[54]  Shane K. Mitchell,et al.  Hydraulically amplified self-healing electrostatic actuators with muscle-like performance , 2018, Science.

[55]  M. Hallett,et al.  Consensus Statement on the classification of tremors. from the task force on tremor of the International Parkinson and Movement Disorder Society , 2018, Movement disorders : official journal of the Movement Disorder Society.

[56]  Max Juraschek,et al.  Mixed Reality in Learning Factories , 2018 .

[57]  Seokhee Jeon,et al.  Pneumatic Actuation in Haptic-Enabled Medical Simulators: A Review , 2018, IEEE Access.

[58]  Ravinder Dahiya,et al.  Robotic Tactile Perception of Object Properties: A Review , 2017, ArXiv.

[59]  Claudio Pacchierotti,et al.  Evaluation of Wearable Haptic Systems for the Fingers in Augmented Reality Applications , 2017, IEEE Transactions on Haptics.

[60]  Clément Gosselin,et al.  Design, analysis and experimental validation of an ungrounded haptic interface using a piezoelectric actuator , 2017 .

[61]  Olga Troynikov,et al.  Evaluation of Flexible Force Sensors for Pressure Monitoring in Treatment of Chronic Venous Disorders , 2017, Sensors.

[62]  Y. Hao,et al.  Electronic Devices for Human‐Machine Interfaces , 2017 .

[63]  Zheng Lou,et al.  Polymer‐Enhanced Highly Stretchable Conductive Fiber Strain Sensor Used for Electronic Data Gloves , 2016 .

[64]  Massimo De Vittorio,et al.  AlN-based flexible piezoelectric skin for energy harvesting from human motion , 2016 .

[65]  Qing Wang,et al.  Toward Wearable Cooling Devices: Highly Flexible Electrocaloric Ba0.67Sr0.33TiO3 Nanowire Arrays , 2016, Advanced materials.

[66]  Nicholas V. Annetta,et al.  Restoring cortical control of functional movement in a human with quadriplegia , 2016, Nature.

[67]  Domenico Prattichizzo,et al.  Haptic Assistive Bracelets for Blind Skier Guidance , 2016, AH.

[68]  Kapil D. Katyal,et al.  Individual finger control of a modular prosthetic limb using high-density electrocorticography in a human subject , 2016, Journal of neural engineering.

[69]  Dongyan Shi,et al.  A unified method for free vibration analysis of circular, annular and sector plates with arbitrary boundary conditions , 2016 .

[70]  Lasse Rosendahl,et al.  A comparison of micro-structured flat-plate and cross-cut heat sinks for thermoelectric generation application , 2015 .

[71]  I. Carlsson,et al.  Forearm amputees' views of prosthesis use and sensory feedback. , 2015, Journal of hand therapy : official journal of the American Society of Hand Therapists.

[72]  Roberto Manduchi,et al.  Vibrotactile Guidance for Wayfinding of Blind Walkers , 2015, IEEE Transactions on Haptics.

[73]  Peter H Veltink,et al.  Vibrotactile grasping force and hand aperture feedback for myoelectric forearm prosthesis users , 2015, Prosthetics and orthotics international.

[74]  Mari Velonaki,et al.  Robotics and Autonomous Systems , 2014 .

[75]  Hiroshi Ishii,et al.  Physical telepresence: shape capture and display for embodied, computer-mediated remote collaboration , 2014, UIST.

[76]  Tomasz Starecki,et al.  Analog Front-End Circuitry in Piezoelectric and Microphone Detection of Photoacoustic Signals , 2014 .

[77]  X. Moya,et al.  Caloric materials near ferroic phase transitions. , 2014, Nature materials.

[78]  Halit Eren,et al.  Measurement, Instrumentation, and Sensors Handbook, Second Edition - 2 Volume Set , 2014 .

[79]  Robert Oven,et al.  Modified Charge Amplifier for Stray Immune Capacitance Measurements , 2014, IEEE Transactions on Instrumentation and Measurement.

[80]  Sriram Subramanian,et al.  UltraHaptics: multi-point mid-air haptic feedback for touch surfaces , 2013, UIST.

[81]  Octavian Postolache,et al.  Tactile Sensors for Robotic Applications , 2013 .

[82]  Erlend Fagertun Hofstad,et al.  Perceiving haptic feedback in virtual reality simulators , 2013, Surgical Endoscopy.

[83]  Gesche Joost,et al.  Mobile Lorm Glove: introducing a communication device for deaf-blind people , 2012, TEI.

[84]  Aude Bolopion,et al.  Haptic Teleoperation for 3-D Microassembly of Spherical Objects , 2012, IEEE/ASME Transactions on Mechatronics.

[85]  O. Witte,et al.  Sensory feedback prosthesis reduces phantom limb pain: Proof of a principle , 2012, Neuroscience Letters.

[86]  Sachin Chitta,et al.  Human-Inspired Robotic Grasp Control With Tactile Sensing , 2011, IEEE Transactions on Robotics.

[87]  C. Jang,et al.  A Survey on Activities of Daily Living and Occupations of Upper Extremity Amputees , 2011, Annals of rehabilitation medicine.

[88]  Ali Israr,et al.  Tactile brush: drawing on skin with a tactile grid display , 2011, CHI.

[89]  Z. Suo Theory of dielectric elastomers , 2010 .

[90]  Giulio Sandini,et al.  Tactile Sensing—From Humans to Humanoids , 2010, IEEE Transactions on Robotics.

[91]  Rongrong Wang,et al.  Touch & talk: contextualizing remote touch for affective interaction , 2010, TEI '10.

[92]  Tomohiro Amemiya,et al.  Haptic handheld wayfinder with pseudo-attraction force for pedestrians with visual impairments , 2009, Assets '09.

[93]  Taro Maeda,et al.  Directional Force Sensation by Asymmetric Oscillation From a Double-Layer Slider-Crank Mechanism , 2009, J. Comput. Inf. Sci. Eng..

[94]  Tomohiro Amemiya,et al.  Lead-me interface for a pulling sensation from hand-held devices , 2008, TAP.

[95]  Nicholas Caporusso,et al.  A wearable Malossi alphabet interface for deafblind people , 2008, AVI '08.

[96]  Gabrielle H. Saunders,et al.  An Overview of Dual Sensory Impairment in Older Adults: Perspectives for Rehabilitation , 2007, Trends in amplification.

[97]  Masahiko Inami,et al.  Remote active tangible interactions , 2007, TEI.

[98]  James R. Marston,et al.  Personal Guidance System for People with Visual Impairment: A Comparison of Spatial Displays for Route Guidance , 2005, Journal of visual impairment & blindness.

[99]  Uwe D. Hanebeck,et al.  Design issues of mobile haptic interfaces , 2003, J. Field Robotics.

[100]  Kenneth O. Johnson,et al.  The roles and functions of cutaneous mechanoreceptors , 2001, Current Opinion in Neurobiology.

[101]  Frank Biocca,et al.  The Cyborg's Dilemma: Progressive Embodiment in Virtual Environments , 2006, J. Comput. Mediat. Commun..

[102]  C J Vierck,et al.  A critical band filter in touch , 1995, The Journal of neuroscience : the official journal of the Society for Neuroscience.

[103]  Roger Giblin,et al.  Hand-tapper III: a prototype communication device using finger-spelling , 1991 .

[104]  W.J. Tompkins,et al.  Electrotactile and vibrotactile displays for sensory substitution systems , 1991, IEEE Transactions on Biomedical Engineering.

[105]  H.J. Chizeck,et al.  Feedback regulation of hand grasp opening and contact force during stimulation of paralyzed muscle , 1991, IEEE Transactions on Biomedical Engineering.

[106]  R. Johansson,et al.  Properties of cutaneous mechanoreceptors in the human hand related to touch sensation. , 1984, Human neurobiology.