Evaluation of convolutional neural networks for the classification of falls from heterogeneous thermal vision sensors

The automatic detection of falls within environments where sensors are deployed has attracted considerable research interest due to the prevalence and impact of falling people, especially the elderly. In this work, we analyze the capabilities of non-invasive thermal vision sensors to detect falls using several architectures of convolutional neural networks. First, we integrate two thermal vision sensors with different capabilities: (1) low resolution with a wide viewing angle and (2) high resolution with a central viewing angle. Second, we include fuzzy representation of thermal information. Third, we enable the generation of a large data set from a set of few images using ad hoc data augmentation, which increases the original data set size, generating new synthetic images. Fourth, we define three types of convolutional neural networks which are adapted for each thermal vision sensor in order to evaluate the impact of the architecture on fall detection performance. The results show encouraging performance in single-occupancy contexts. In multiple occupancy, the low-resolution thermal vision sensor with a wide viewing angle obtains better performance and reduction of learning time, in comparison with the high-resolution thermal vision sensors with a central viewing angle.

[1]  Xinrong Li,et al.  Wireless Sensor Network System Design Using Raspberry Pi and Arduino for Environmental Monitoring Applications , 2014, FNC/MobiSPC.

[2]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Macarena Espinilla,et al.  Fuzzy Linguistic Protoforms to Summarize Heart Rate Streams of Patients with Ischemic Heart Disease , 2019, Complex..

[4]  Kai-Kuang Ma,et al.  Fuzzy color histogram and its use in color image retrieval , 2002, IEEE Trans. Image Process..

[5]  Hironobu Fujiyoshi,et al.  Improving Quality of Training Samples Through Exhaustless Generation and Effective Selection for Deep Convolutional Neural Networks , 2015, VISAPP.

[6]  Thambipillai Srikanthan,et al.  Vision-based patient monitoring: a comprehensive review of algorithms and technologies , 2018, J. Ambient Intell. Humaniz. Comput..

[7]  Etienne E. Kerre,et al.  Fuzzy Two-Step Filter for Impulse Noise Reduction From Color Images , 2006, IEEE Transactions on Image Processing.

[8]  Ling Shao,et al.  A survey on fall detection: Principles and approaches , 2013, Neurocomputing.

[9]  Kang Ryoung Park,et al.  Pedestrian Detection Based on Adaptive Selection of Visible Light or Far-Infrared Light Camera Image by Fuzzy Inference System and Convolutional Neural Network-Based Verification , 2017, Sensors.

[10]  Mukhtiar Memon,et al.  Ambient Assisted Living Healthcare Frameworks, Platforms, Standards, and Quality Attributes , 2014, Sensors.

[11]  Jesse Hoey,et al.  Sensor-Based Activity Recognition , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[12]  Danielsen Asbjørn,et al.  Recognizing Bedside Events Using Thermal and Ultrasonic Readings , 2017, Sensors.

[13]  Chris D. Nugent,et al.  Computer Vision-Based Gait Velocity from Non-Obtrusive Thermal Vision Sensors , 2018, 2018 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[14]  Shin-Jye Lee,et al.  Image Classification Based on the Boost Convolutional Neural Network , 2018, IEEE Access.

[15]  Neil A. Thacker,et al.  Design of a Visual System for Detecting Natural Events by the use of an Independent Visual Estimate: A Human Fall Detector , 2002 .

[16]  Holly Hedegaard,et al.  Deaths from unintentional injury among adults aged 65 and over: United States, 2000-2013. , 2015, NCHS data brief.

[17]  Jana Kosecka,et al.  Synthesizing Training Data for Object Detection in Indoor Scenes , 2017, Robotics: Science and Systems.

[18]  Miguel Hernando,et al.  Home Camera-Based Fall Detection System for the Elderly , 2017, Sensors.

[19]  C. Nugent,et al.  The Experience of Developing the UJAmI Smart Lab , 2018, IEEE Access.

[20]  Chris D. Nugent,et al.  Detection of Falls from Non-Invasive Thermal Vision Sensors Using Convolutional Neural Networks , 2018, UCAmI.

[21]  Luca Maria Gambardella,et al.  High-Performance Neural Networks for Visual Object Classification , 2011, ArXiv.

[22]  Víctor M. González Suárez,et al.  Improving Fall Detection Using an On-Wrist Wearable Accelerometer , 2018, Sensors.

[23]  Koji Hashimoto,et al.  The use of thermal ir array sensor for indoor fall detection , 2017, 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[24]  Lorenzo Leija,et al.  Development of a thermographic image instrument using the raspberry Pi embedded system for the study of the diabetic foot , 2018, 2018 IEEE International Instrumentation and Measurement Technology Conference (I2MTC).

[25]  Alex Mihailidis,et al.  A Survey on Ambient-Assisted Living Tools for Older Adults , 2013, IEEE Journal of Biomedical and Health Informatics.

[26]  Jean Meunier,et al.  Robust Video Surveillance for Fall Detection Based on Human Shape Deformation , 2011, IEEE Transactions on Circuits and Systems for Video Technology.

[27]  Gerald Penn,et al.  Convolutional Neural Networks for Speech Recognition , 2014, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[28]  Yann LeCun,et al.  The mnist database of handwritten digits , 2005 .

[29]  Akira Hayashida,et al.  New approach for indoor fall detection by infrared thermal array sensor , 2017, 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS).

[30]  Pai Peng,et al.  Gas Classification Using Deep Convolutional Neural Networks , 2018, Sensors.

[31]  Martial Hebert,et al.  Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[32]  Chris D. Nugent,et al.  SensorCentral: A Research Oriented, Device Agnostic, Sensor Data Platform , 2017, UCAmI.

[33]  C. Nugent,et al.  Experiences in the development of a Smart Lab , 2009 .

[34]  Ye Zhang,et al.  Large Scale Remote Sensing Image Segmentation Based on Fuzzy Region Competition and Gaussian Mixture Model , 2018, IEEE Access.

[35]  P. Jonathon Phillips,et al.  Empirical Evaluation Methods in Computer Vision , 2002 .

[36]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[37]  Etienne E. Kerre,et al.  A New Fuzzy Color Correlated Impulse Noise Reduction Method , 2007, IEEE Transactions on Image Processing.

[38]  Tzyy-Ping Jung,et al.  Fall Prediction and Prevention Systems: Recent Trends, Challenges, and Future Research Directions , 2017, Sensors.

[39]  Perianayagam Arokiasamy,et al.  Prevalence, risk factors and disability associated with fall-related injury in older adults in low- and middle-incomecountries: results from the WHO Study on global AGEing and adult health (SAGE) , 2015, BMC Medicine.

[40]  Jacques Demongeot,et al.  A Novel Monitoring System for Fall Detection in Older People , 2018, IEEE Access.

[41]  Sergey Ioffe,et al.  Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning , 2016, AAAI.

[42]  Josef Bigün,et al.  PastVision+: Thermovisual Inference of Recent Medicine Intake by Detecting Heated Objects and Cooled Lips , 2017, Front. Robot. AI.

[43]  Shaowen Yao,et al.  A Novel Multi-Focus Image Fusion Method Based on Stationary Wavelet Transform and Local Features of Fuzzy Sets , 2017, IEEE Access.