Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays

Signed languages are not as pervasive a conversational medium as spoken languages due to the history of institutional suppression of the former and the linguistic hegemony of the latter. This has led to a communication barrier between signers and non-signers that could be mitigated by technology-mediated approaches. Here, we show that a wearable sign-to-speech translation system, assisted by machine learning, can accurately translate the hand gestures of American Sign Language into speech. The wearable sign-to-speech translation system is composed of yarn-based stretchable sensor arrays and a wireless printed circuit board, and offers a high sensitivity and fast response time, allowing real-time translation of signs into spoken words to be performed. By analysing 660 acquired sign language hand gesture recognition patterns, we demonstrate a recognition rate of up to 98.63% and a recognition time of less than 1 s. Wearable yarn-based stretchable sensor arrays, combined with machine learning, can be used to translate American Sign Language into speech in real time.

[1]  Yihui Zhang,et al.  Binodal, wireless epidermal electronic systems with in-sensor analytics for neonatal intensive care , 2019, Science.

[2]  Derek Ho,et al.  Glove-based hand gesture recognition sign language translator using capacitive touch sensor , 2016, 2016 IEEE International Conference on Electron Devices and Solid-State Circuits (EDSSC).

[3]  Zhibin Yu,et al.  User-interactive electronic skin for instantaneous pressure visualization. , 2013, Nature materials.

[4]  Alex Pentland,et al.  Real-Time American Sign Language Recognition Using Desk and Wearable Computer Based Video , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[5]  Boris Murmann,et al.  Skin electronics from scalable fabrication of an intrinsically stretchable transistor array , 2018, Nature.

[6]  Benjamin C. K. Tee,et al.  Skin-like pressure and strain sensors based on transparent elastic films of carbon nanotubes. , 2011, Nature nanotechnology.

[7]  Zhong Lin Wang,et al.  Taxel-Addressable Matrix of Vertical-Nanowire Piezotronic Transistors for Active and Adaptive Tactile Imaging , 2013, Science.

[8]  Long Jin,et al.  A linear-to-rotary hybrid nanogenerator for high-performance wearable biomechanical energy harvesting , 2020 .

[9]  Sanat S Bhole,et al.  Soft Microfluidic Assemblies of Sensors, Circuits, and Radios for the Skin , 2014, Science.

[10]  M. Kaltenbrunner,et al.  An ultra-lightweight design for imperceptible plastic electronics , 2013, Nature.

[11]  Ferat Sahin,et al.  Real-Time American Sign Language Recognition System Using Surface EMG Signal , 2015, 2015 IEEE 14th International Conference on Machine Learning and Applications (ICMLA).

[12]  Xue Wang,et al.  A Wireless Textile-Based Sensor System for Self-Powered Personalized Health Care , 2020 .

[13]  B. Prabhakaran,et al.  Hand-Gesture Computing for the Hearing and Speech Impaired , 2008, IEEE MultiMedia.

[14]  Haiwen Luan,et al.  Skin-integrated wireless haptic interfaces for virtual and augmented reality , 2019, Nature.

[15]  Jun Chen,et al.  Smart Textiles for Electricity Generation. , 2020, Chemical reviews.

[16]  Zhong Lin Wang,et al.  Progress in triboelectric nanogenerators as a new energy technology and self-powered sensors , 2015 .

[17]  Youngoh Lee,et al.  Transparent and conductive nanomembranes with orthogonal silver nanowire arrays for skin-attachable loudspeakers and microphones , 2018, Science Advances.

[18]  Vasiliki Kosmidou,et al.  Sign Language Recognition Using Intrinsic-Mode Sample Entropy on sEMG and Accelerometer Data , 2009, IEEE Transactions on Biomedical Engineering.

[19]  Zhong Lin Wang,et al.  Flexible triboelectric generator , 2012 .

[20]  Boon Giin Lee,et al.  Smart Wearable Hand Device for Sign Language Interpretation System With Sensors Fusion , 2018, IEEE Sensors Journal.

[21]  Muhammad Mahadi Abdul Jamil,et al.  Development of a Wearable Device for Sign Language Recognition , 2018, Journal of Physics: Conference Series.

[22]  I. Park,et al.  Highly stretchable and sensitive strain sensor based on silver nanowire-elastomer nanocomposite. , 2014, ACS nano.

[23]  Michael C. McAlpine,et al.  Highly ordered nanowire arrays on plastic substrates for ultrasensitive flexible chemical sensors. , 2007, Nature materials.

[24]  Takao Someya,et al.  Stretchable and waterproof elastomer-coated organic photovoltaics for washable electronic textile applications , 2017 .

[25]  Nannan Zhang,et al.  Micro-cable structured textile for simultaneously harvesting solar and mechanical energy , 2016, Nature Energy.

[26]  Jun Chen,et al.  Single-layered ultra-soft washable smart textiles for all-around ballistocardiograph, respiration, and posture monitoring during sleep. , 2020, Biosensors & bioelectronics.

[27]  Federico Sandoval-Ibarra,et al.  American Sign Language Alphabet Recognition Using a Neuromorphic Sensor and an Artificial Neural Network , 2017, Sensors.

[28]  Raeed H. Chowdhury,et al.  Epidermal Electronics , 2011, Science.

[29]  Zhong Lin Wang,et al.  Reviving Vibration Energy Harvesting and Self-Powered Sensing by a Triboelectric Nanogenerator , 2017 .

[30]  Timothy F. O'Connor,et al.  The Language of Glove: Wireless gesture decoder with low-power and stretchable hybrid electronics , 2017, PloS one.

[31]  Yi Nie,et al.  Photo-Rechargeable Fabrics as Sustainable and Robust Power Sources for Wearable Bioelectronics , 2020 .

[32]  K. Hata,et al.  A stretchable carbon nanotube strain sensor for human-motion detection. , 2011, Nature nanotechnology.

[33]  Takao Someya,et al.  Inflammation-free, gas-permeable, lightweight, stretchable on-skin electronics with nanomeshes. , 2017, Nature nanotechnology.

[34]  Cunjiang Yu,et al.  Fully rubbery integrated electronics from high effective mobility intrinsically stretchable semiconductors , 2019, Science Advances.

[35]  Sam Emaminejad,et al.  Fully integrated wearable sensor arrays for multiplexed in situ perspiration analysis , 2016, Nature.

[36]  Zhenan Bao,et al.  A stretchable and biodegradable strain and pressure sensor for orthopaedic application , 2018 .

[37]  Sheng Xu,et al.  Three-dimensional integrated stretchable electronics , 2018, Nature Electronics.

[38]  Munib Qutaishat,et al.  American sign language (ASL) recognition based on Hough transform and neural networks , 2007, Expert Syst. Appl..

[39]  Sunil Vadera,et al.  A convolutional neural network to classify American Sign Language fingerspelling from depth and colour images , 2017, Expert Syst. J. Knowl. Eng..

[40]  Hao Liu,et al.  Passive and Space-Discriminative Ionic Sensors Based on Durable Nanocomposite Electrodes toward Sign Language Recognition. , 2017, ACS nano.

[41]  Qifa Zhou,et al.  Monitoring of the central blood pressure waveform via a conformal ultrasonic device , 2018, Nature Biomedical Engineering.

[42]  Jun Zhou,et al.  High‐Strain Sensors Based on ZnO Nanowire/Polystyrene Hybridized Flexible Films , 2011, Advanced materials.

[43]  Roozbeh Jafari,et al.  A Wearable System for Recognizing American Sign Language in Real-Time Using IMU and Surface EMG Sensors , 2016, IEEE Journal of Biomedical and Health Informatics.