Automated Non-Contact Respiratory Rate Monitoring of Neonates Based on Synchronous Evaluation of a 3D Time-of-Flight Camera and a Microwave Interferometric Radar Sensor

This paper introduces an automatic non-contact monitoring method based on the synchronous evaluation of a 3D time-of-flight (ToF) camera and a microwave interferometric radar sensor for measuring the respiratory rate of neonates. The current monitoring on the Neonatal Intensive Care Unit (NICU) has several issues which can cause pressure marks, skin irritations and eczema. To minimize these risks, a non-contact system made up of a 3D time-of-flight camera and a microwave interferometric radar sensor is presented. The 3D time-of-flight camera delivers 3D point clouds which can be used to calculate the change in distance of the moving chest and from it the respiratory rate. The disadvantage of the ToF camera is that the heartbeat cannot be determined. The microwave interferometric radar sensor determines the change in displacement caused by the respiration and is even capable of measuring the small superimposed movements due to the heartbeat. The radar sensor is very sensitive towards movement artifacts due to, e.g., the baby moving its arms. To allow a robust vital parameter detection the data of both sensors was evaluated synchronously. In this publication, we focus on the first step: determining the respiratory rate. After all processing steps, the respiratory rate determined by the radar sensor was compared to the value received from the 3D time-of-flight camera. The method was validated against our gold standard: a self-developed neonatal simulation system which can simulate different breathing patterns. In this paper, we show that we are the first to determine the respiratory rate by evaluating the data of an interferometric microwave radar sensor and a ToF camera synchronously. Our system delivers very precise breaths per minute (BPM) values within the norm range of 20–60 BPM with a maximum difference of 3 BPM (for the ToF camera itself at 30 BPM in normal mode). Especially in lower respiratory rate regions, i.e., 5 and 10 BPM, the synchronous evaluation is required to compensate the drawbacks of the ToF camera. In the norm range, the ToF camera performs slightly better than the radar sensor.

[1]  A Koelpin,et al.  The Six-Port in Modern Society , 2010, IEEE Microwave Magazine.

[2]  H. Schumacher,et al.  IR-UWB Radar Demonstrator for Ultra-Fine Movement Detection and Vital-Sign Monitoring , 2013, IEEE Transactions on Microwave Theory and Techniques.

[3]  Thomas Penzel,et al.  3D Camera and Pulse Oximeter for Respiratory Events Detection , 2020, IEEE Journal of Biomedical and Health Informatics.

[4]  G. Piacentini,et al.  Structured Light Plethysmography (SLP): Management and follow up of a paediatric patient with pneumonia , 2017, Respiratory medicine case reports.

[5]  Robert Weigel,et al.  A contactless system for continuous vital sign monitoring in palliative and intensive care , 2018, 2018 Annual IEEE International Systems Conference (SysCon).

[6]  Abbas K. Abbas,et al.  Neonatal non-contact respiratory monitoring based on real-time infrared thermography , 2011, Biomedical engineering online.

[7]  Sander Stuijk,et al.  Camera-Based On-Line Short Cessation of Breathing Detection , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[8]  Robert Lange,et al.  3D time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD-technology , 2006 .

[9]  Tzyy-Sheng Horng,et al.  A Self- and Mutually Injection-Locked Radar System for Monitoring Vital Signs in Real Time With Random Body Movement Cancellation , 2016, IEEE Transactions on Microwave Theory and Techniques.

[10]  Albert van der Veen,et al.  Development of a baby friendly non-contact method for measuring vital signs: First results of clinical measurements in an open incubator at a neonatal intensive care unit , 2014, Photonics West - Biomedical Optics.

[11]  Nico Blodow,et al.  Towards 3D Point cloud based object maps for household environments , 2008, Robotics Auton. Syst..

[12]  Min-Hyung Choi,et al.  WiKiSpiro: non-contact respiration volume monitoring during sleep , 2016, S3@MobiCom.

[13]  Soon-Yong Park,et al.  Real-Time External Respiratory Motion Measuring Technique Using an RGB-D Camera and Principal Component Analysis † , 2017, Sensors.

[14]  P. A. Gorry.  General least-squares smoothing and differentiation by the convolution (Savitzky-Golay) method , 1990 .

[15]  Remo Sala,et al.  A Survey on 3D Cameras: Metrological Comparison of Time-of-Flight, Structured-Light and Active Stereoscopy Technologies , 2018, SpringerBriefs in Computer Science.

[16]  Changzhi Li,et al.  Verification of a non-contact vital sign monitoring system using an infant simulator , 2009, 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[17]  Alexander Koelpin,et al.  Automatic non-contact monitoring of the respiratory rate of neonates using a structured light camera , 2020, 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC).

[18]  E. Chadwick,et al.  Tidal breathing parameters measured by structured light plethysmography in children aged 2–12 years recovering from acute asthma/wheeze compared with healthy children , 2018, Physiological reports.

[19]  Andreas R. Diewald,et al.  Verification and first test measurement of a microwave-based vital sign monitor , 2019 .

[20]  Sandy Rihana,et al.  Kinect2 — Respiratory movement detection study , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[21]  E. P. Tomasini,et al.  An optical measurement method for the simultaneous assessment of respiration and heart rates in preterm infants. , 2013, The Review of scientific instruments.

[22]  Alexander Koelpin,et al.  Microwave interferometer radar-based vital sign detection for driver monitoring syst , 2015, 2015 IEEE MTT-S International Conference on Microwaves for Intelligent Mobility (ICMIM).

[23]  Ales Procházka,et al.  Microsoft Kinect Visual and Depth Sensors for Breathing and Heart Rate Analysis , 2016, Sensors.

[24]  G. Ramachandran,et al.  Three-dimensional reconstruction of cardiac displacement patterns on the chest wall during the P, QRS and T-segments of the ECG by laser speckle inteferometry , 1989, Medical and Biological Engineering and Computing.

[25]  Frédo Durand,et al.  Eulerian video magnification for revealing subtle changes in the world , 2012, ACM Trans. Graph..

[26]  Rainer Stiefelhagen,et al.  Breathing Rate Monitoring during Sleep from a Depth Camera under Real-Life Conditions , 2017, 2017 IEEE Winter Conference on Applications of Computer Vision (WACV).

[27]  Steffen Leonhardt,et al.  Hybrid optical imaging technology for long-term remote monitoring of skin perfusion and temperature behavior , 2014, Journal of biomedical optics.

[28]  Steffen Leonhardt,et al.  A novel ultra-wideband 80 GHz FMCW radar system for contactless monitoring of vital signs , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[29]  Ali Al-Naji,et al.  Remote respiratory monitoring system based on developing motion magnification technique , 2016, Biomed. Signal Process. Control..

[30]  Lorenzo Scalise,et al.  Non contact measurement of heart and respiration rates based on Kinect™ , 2014, 2014 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[31]  Paolo Bernardi,et al.  Comparison between UWB and CW radar sensors for breath activity monitoring , 2014, Defense + Security Symposium.

[32]  Robert Weigel,et al.  Six-Port Radar Sensor for Remote Respiration Rate and Heartbeat Vital-Sign Monitoring , 2013, IEEE Transactions on Microwave Theory and Techniques.

[33]  Wassim Bouachir,et al.  A computer vision method for respiratory monitoring in intensive care environment using RGB-D cameras , 2017, 2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA).

[34]  Wassim Bouachir,et al.  Computerized Medical Imaging and Graphics 3 D imaging system for respiratory monitoring in pediatric intensive care environment , 2018 .

[35]  S. Longhi,et al.  Respiratory rate detection algorithm based on RGB-D camera: theoretical background and experimental results. , 2014, Healthcare technology letters.

[36]  João Jorge,et al.  Non-Contact Monitoring of Respiration in the Neonatal Intensive Care Unit , 2017, 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017).

[37]  Steffen Leonhardt,et al.  Noncontact Monitoring of Respiratory Rate in Newborn Infants Using Thermal Imaging , 2019, IEEE Transactions on Biomedical Engineering.

[38]  Emanuele Frontoni,et al.  Non-Contact Monitoring of Preterm Infants Using RGB-D Camera , 2015 .

[39]  Andreas R. Diewald,et al.  Verification of Algorithm for an I/Q-Radar System for Breathing Detection in an Incubator , 2019, 2019 International Conference on Electromagnetics in Advanced Applications (ICEAA).

[40]  Young-Hyo Lim,et al.  Non-contact respiration monitoring using impulse radio ultrawideband radar in neonates , 2019, Royal Society Open Science.

[41]  Robert Weigel,et al.  A Compact, Versatile Six-Port Radar Module for Industrial and Medical Applications , 2013, J. Electr. Comput. Eng..

[42]  Joachim Hornegger,et al.  Robust real-time 3D respiratory motion detection using time-of-flight cameras , 2008, International Journal of Computer Assisted Radiology and Surgery.

[43]  Michael Arens,et al.  Learning and Tracking the 3D Body Shape of Freely Moving Infants from RGB-D sequences , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[44]  B. Cooper,et al.  Mini review shows that structured light plethysmography provides a non‐contact method for evaluating breathing patterns in children , 2019, Acta paediatrica.

[45]  Remo Sala,et al.  State-of-the-Art Devices Comparison , 2018 .

[46]  Steffen Leonhardt,et al.  Estimation of respiratory rate from thermal videos of preterm infants , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[47]  Robert Weigel,et al.  Advanced template matching algorithm for instantaneous heartbeat detection using continuous wave radar systems , 2017, 2017 First IEEE MTT-S International Microwave Bio Conference (IMBIOC).

[48]  G. Matthews,et al.  A non-contact vital signs monitor. , 2000, Critical reviews in biomedical engineering.

[49]  E. Graves,et al.  Radar remote monitoring of vital signs , 2009, IEEE Microwave Magazine.