Apollo SignSound: an intelligent system applied to ubiquitous healthcare of deaf people

In Brazil, there are more than 9.3 million people with Hearing Impairment (PWHI) and deaf who face daily accessibility difficulties. On the other hand, there is the growth of the use of mobile devices and the application of the Internet of Things. The motivation for the development of this work lies in the absence of user interface systems based on sign language and customizable according to the profile that is applied to the prevention of risks external to the health of the deaf. This paper proposes the Apollo SignSound system, which promotes accessibility for PWHI and deaf people in a smart home environment, especially regarding safety. The scientific contribution of Apollo SignSound lies in the detection of ambient risks using neural networks. Besides this, SignSound also considers the user profile, mainly the degree of deafness, to generate accessible notifications represented in Brazilian Sign Language (LIBRAS in Portuguese). The notifications of risk sent to the smartphone of PWHI or deaf also can vibrate or turn on the light of the device. We implemented a prototype of a smart home that collects environmental sounds and notifies the deaf user. The scenario-based assessment included three activities of daily living events of a deaf user: a kettle boiling over the stove, a dog barking, and one person knocking on the door. The results indicate the means of the f score of 0.73 for accuracy evaluation. Usability and acceptance evaluations were performed by five deaf users and the results indicate the approval of 90% in the perceived ease of use and perceived usefulness.

[1]  Mark Weiser,et al.  The computer for the 21st Century , 1991, IEEE Pervasive Computing.

[2]  Jorge L. V. Barbosa,et al.  Gamification and serious games in depression care: A systematic mapping study , 2018, Telematics Informatics.

[3]  S. R. Silverman,et al.  Hearing and deafness , 1970 .

[4]  Jorge L. V. Barbosa,et al.  A model for learning objects adaptation in light of mobile and context-aware computing , 2016, Personal and Ubiquitous Computing.

[6]  Débora Nice Ferrari Barbosa,et al.  Learning in Ubiquitous Computing Environments , 2012, Int. J. Inf. Commun. Technol. Educ..

[7]  Hyoung-Gook Kim,et al.  Sound Event Detection and Haptic Vibration Based Home Monitoring Assistant System for the Deaf and Hard-of-Hearing , 2018, Proceedings of the 2018 Workshop on Multimedia for Accessible Human Computer Interface.

[8]  Jorge Luis Victória Barbosa,et al.  Using Learner Group Profiles for Content Recommendation in Ubiquitous Environments , 2020, Int. J. Inf. Commun. Technol. Educ..

[9]  David K. McGookin,et al.  Investigating user generated presentations of self in face-to-face interaction between strangers , 2017, Int. J. Hum. Comput. Stud..

[10]  Jorge L. V. Barbosa,et al.  MUCS: A model for ubiquitous commerce support , 2011, Electron. Commer. Res. Appl..

[11]  Anind K. Dey,et al.  Understanding and Using Context , 2001, Personal and Ubiquitous Computing.

[12]  Jorge L. V. Barbosa,et al.  A computational model for soil fertility prediction in ubiquitous agriculture , 2020, Comput. Electron. Agric..

[13]  Cheolho Yoon,et al.  Convenience and TAM in a ubiquitous computing environment: The case of wireless LAN , 2007, Electron. Commer. Res. Appl..

[14]  Gregg C. Vanderheiden,et al.  Ubiquitous Accessibility, Common Technology Core, and Micro Assistive Technology: Commentary on “Computers and People with Disabilities” , 2008, TACC.

[15]  Jorge Luis Victória Barbosa,et al.  A scalable model for building context-aware applications for noncommunicable diseases prevention , 2019, Inf. Process. Lett..

[16]  Débora Nice Ferrari Barbosa,et al.  Content management in a ubiquitous learning environment , 2013, Int. J. Comput. Appl. Technol..

[17]  Kristrun Gunnarsdottir,et al.  Ambient intelligence : a narrative in search of users (discussion paper) , 2011 .

[18]  Bruno Alves,et al.  TrailCare: An indoor and outdoor Context-aware system to assist wheelchair users , 2018, Int. J. Hum. Comput. Stud..

[19]  Débora Nice Ferrari Barbosa,et al.  A model for profile management applied to ubiquitous learning environments , 2014, Expert Syst. Appl..

[20]  Jorge Luis Victória Barbosa,et al.  Ubiquitous computing: Applications and research opportunities , 2015 .

[21]  Mahadev Satyanarayanan,et al.  Pervasive computing: vision and challenges , 2001, IEEE Wirel. Commun..

[22]  Francys Tadeu Ribeiro.  Internet das coisas : da teoria à prática. , 2019 .

[23]  R. Likert.  “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[24]  Mark Weiser.  The computer for the 21st century , 1991 .

[25]  J. Barbosa,et al.  Hermes: um modelo para acessibilidade ubíqua dedicado à deficiência auditiva , 2016 .

[26]  Jorge L. V. Barbosa,et al.  A Model for Ubiquitous Care of Noncommunicable Diseases , 2014, IEEE Journal of Biomedical and Health Informatics.

[27]  Fulvio Corno,et al.  Design and Development of One-Switch Video Games for Children with Severe Motor Disabilities , 2017, ACM Trans. Access. Comput..

[28]  Jorge L. V. Barbosa,et al.  An intelligent system for prognosis of noncommunicable diseases' risk factors , 2018, Telematics Informatics.

[29]  Jorge L. V. Barbosa,et al.  A ubiquitous learning model focused on learner interaction , 2011, Int. J. Learn. Technol..

[30]  Jorge L. V. Barbosa,et al.  Hefestos: an intelligent system applied to ubiquitous accessibility , 2016, Universal Access in the Information Society.

[31]  Fariba Sadri,et al.  Ambient intelligence: A survey , 2011, CSUR.

[32]  Mieczyslaw M. Kokar,et al.  Ontology-based situation awareness , 2009, Inf. Fusion.

[33]  Débora Nice Ferrari Barbosa,et al.  TrailTrade: A model for trail-aware commerce support , 2016, Comput. Ind..

[34]  Rodrigo Bavaresco,et al.  Design and evaluation of a context-aware model based on psychophysiology , 2019, Comput. Methods Programs Biomed..

[35]  Jorge L. V. Barbosa,et al.  Pompilos, a Model for Augmenting Health Assistant Applications with Social Media Content , 2020, J. Univers. Comput. Sci..

[36]  Mica R. Endsley,et al.  Toward a Theory of Situation Awareness in Dynamic Systems , 1995, Hum. Factors.

[37]  Débora Nice Ferrari Barbosa,et al.  Applied Computing to Education on Noncommunicable Chronic Diseases: A Systematic Mapping Study. , 2020, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.