Real-time multimodal 3D imaging system for remote estimation of vital signs

The contactless estimation of vital signs based on conventional color cameras and ambient light can be affected by motions of patients and changes in ambient light. In this work, a multimodal 3D imaging system with an irritation-free controlled illumination was developed in order to mitigate these both problems. In the developed system the real-time high-precision 3D imaging is combined with VIS-NIR multispectral imaging and thermal imaging. Based on 3D data and color images, an approach was proposed for the efficient compensation of head motions, and novel approaches based on 3D regions of interest were developed for the estimation of heart rate, oxygen saturation, respiration rate, and body temperature from NIR multispectral video data and thermal video data. A proof-of-concept for the developed imaging system and algorithms can be delivered with first experimental results.

[1]  Andreas Tünnermann,et al.  5D hyperspectral imaging: fast and accurate measurement of surface shape and spectral characteristics using structured light. , 2018, Optics express.

[2]  Ayoub Al-Hamadi,et al.  How the Region of Interest Impacts Contact Free Heart Rate Estimation Algorithms , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[3]  Ki H. Chon,et al.  Physiological Parameter Monitoring from Optical Recordings With a Mobile Phone , 2012, IEEE Transactions on Biomedical Engineering.

[4]  Andreas Tünnermann,et al.  GOBO projection for 3D measurements at highest frame rates: a performance analysis , 2018, Light: Science & Applications.

[5]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[6]  Sander Stuijk,et al.  Robust respiration detection from remote photoplethysmography. , 2016, Biomedical optics express.

[7]  Celestino Ordóñez,et al.  Detection of human vital signs in hazardous environments by means of video magnification , 2018, PloS one.

[8]  Chen Zhang,et al.  A Multi-Spectral Database for NIR Heart Rate Estimation , 2018, 2018 25th IEEE International Conference on Image Processing (ICIP).

[9]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[10]  Andreas Tünnermann,et al.  High-speed 3D thermography , 2019, Optics and Lasers in Engineering.

[11]  Frédo Durand,et al.  Eulerian video magnification for revealing subtle changes in the world , 2012, ACM Trans. Graph..

[12]  Franck Marzani,et al.  Integration of 3D and multispectral data for cultural heritage applications: Survey and perspectives , 2013, Image Vis. Comput..

[13]  Edward H. Adelson,et al.  PYRAMID METHODS IN IMAGE PROCESSING. , 1984 .

[14]  L. Tarassenko,et al.  Non-contact video-based vital sign monitoring using ambient light and auto-regressive models , 2014, Physiological measurement.

[15]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[16]  Gerard de Haan,et al.  Robust Pulse Rate From Chrominance-Based rPPG , 2013, IEEE Transactions on Biomedical Engineering.

[17]  Ashok Veeraraghavan,et al.  DistancePPG: Robust non-contact vital signs monitoring using a camera , 2015, Biomedical optics express.

[18]  Jian Sun,et al.  Face Alignment at 3000 FPS via Regressing Local Binary Features , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Chen Zhang,et al.  3D high-resolution multimodal imaging system for real-time applications , 2020, Defense + Commercial Sensing.

[20]  D. Altman,et al.  STATISTICAL METHODS FOR ASSESSING AGREEMENT BETWEEN TWO METHODS OF CLINICAL MEASUREMENT , 1986, The Lancet.

[21]  U. Bal.  Non-contact estimation of heart rate and oxygen saturation using ambient light. , 2015, Biomedical optics express.

[22]  Andreas Tünnermann,et al.  Theoretical considerations on aperiodic sinusoidal fringes in comparison to phase-shifted sinusoidal fringes for high-speed three-dimensional shape measurement. , 2015, Applied optics.

[23]  Ig-Jae Kim,et al.  3D Multi-Spectrum Sensor System with Face Recognition , 2013, Sensors.

[24]  Rafik A. Goubran,et al.  Adaptive eulerian video magnification methods to extract heart rate from thermal video , 2016, 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[25]  Lionel Tarassenko,et al.  Non-contact measurement of oxygen saturation with an RGB camera. , 2015, Biomedical optics express.

[26]  Adam Chromy,et al.  A 3D Scan Model and Thermal Image Data Fusion Algorithms for 3D Thermography in Medicine , 2017, Journal of healthcare engineering.

[27]  Shuchang Xu,et al.  Robust efficient estimation of heart rate pulse from video. , 2014, Biomedical optics express.

[28]  Peter Kühmstedt,et al.  BICOS—An Algorithm for Fast Real-Time Correspondence Search for Statistical Pattern Projection-Based Active Stereo Sensors , 2019, Applied Sciences.

[29]  L. O. Svaasand,et al.  Remote plethysmographic imaging using ambient light. , 2008, Optics express.

[30]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[31]  Peter Robinson,et al.  OpenFace: An open source facial behavior analysis toolkit , 2016, 2016 IEEE Winter Conference on Applications of Computer Vision (WACV).

[32]  Ayoub Al-Hamadi,et al.  Fusion-Based Approach for Respiratory Rate Recognition From Facial Video Images , 2020, IEEE Access.

[33]  Steffen Leonhardt,et al.  Remote monitoring of breathing dynamics using infrared thermography. , 2015, Biomedical optics express.

[34]  James R. Green,et al.  Eulerian Magnification of Multi-Modal RGB-D Video for Heart Rate Estimation , 2018, 2018 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[35]  A. Tünnermann,et al.  High-speed three-dimensional shape measurement using GOBO projection , 2016 .

[36]  Nicolai Marquardt,et al.  Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging. , 2017, Biomedical optics express.

[37]  Roberto Cesar Betini,et al.  Noncontact SpO2 Measurement Using Eulerian Video Magnification , 2020, IEEE Transactions on Instrumentation and Measurement.