Preterm Infants’ Pose Estimation With Spatio-Temporal Features

Objective: Preterm infants’ limb monitoring in neonatal intensive care units (NICUs) is of primary importance for assessing infants’ health status and motor/cognitive development. Herein, we propose a new approach to preterm infants’ limb pose estimation that features spatio-temporal information to detect and track limb joints from depth videos with high reliability. Methods: Limb-pose estimation is performed using a deep-learning framework consisting of a detection and a regression convolutional neural network (CNN) for rough and precise joint localization, respectively. The CNNs are implemented to encode connectivity in the temporal direction through 3D convolution. Assessment of the proposed framework is performed through a comprehensive study with sixteen depth videos acquired in the actual clinical practice from sixteen preterm infants (the babyPose dataset). Results: When applied to pose estimation, the median root mean square distance, computed among all limbs, between the estimated and the ground-truth pose was 9.06 pixels, overcoming approaches based on spatial features only (11.27 pixels). Conclusion: Results showed that the spatio-temporal features had a significant influence on the pose-estimation performance, especially in challenging cases (e.g., homogeneous image intensity). Significance: This article significantly enhances the state of art in automatic assessment of preterm infants’ health status by introducing the use of spatio-temporal features for limb detection and tracking, and by being the first study to use depth videos acquired in the actual clinical practice for limb-pose estimation. The babyPose dataset has been released as the first annotated dataset for infants’ pose estimation.

[1]  Fei-Fei Li,et al.  Large-Scale Video Classification with Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[2]  Danail Stoyanov,et al.  Deep Learning Based Robotic Tool Detection and Articulation Estimation With Spatio-Temporal Layers , 2019, IEEE Robotics and Automation Letters.

[3]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Mubarak Shah,et al.  An End-to-end 3D Convolutional Neural Network for Action Detection and Segmentation in Videos , 2017, ArXiv.

[5]  Weiyang Deng,et al.  Development of a Wearable Sensor Algorithm to Detect the Quantity and Kinematic Characteristics of Infant Arm Movement Bouts Produced across a Full Day in the Natural Environment. , 2017, Technologies.

[6]  Øyvind Stavdahl,et al.  Weakly supervised motion segmentation with particle matching , 2015, Comput. Vis. Image Underst..

[7]  C. Heriza,et al.  Comparison of leg movements in preterm infants at term with healthy full-term infants. , 1988, Physical therapy.

[8]  G. Greisen,et al.  [European consensus guidelines on the management of neonatal respiratory distress syndrome in preterm infants--2013 update]. , 2014, Zhonghua er ke za zhi = Chinese journal of pediatrics.

[9]  Steffen Leonhardt,et al.  Noncontact Monitoring of Respiratory Rate in Newborn Infants Using Thermal Imaging , 2019, IEEE Transactions on Biomedical Engineering.

[10]  J. Sweeney,et al.  Musculoskeletal Implications of Preterm Infant Positioning in the NICU , 2002, The Journal of perinatal & neonatal nursing.

[11]  Giovanni Cioni,et al.  Prechtl's Method on the Qualitative Assessment of General Movements in Preterm, Term and Young Infants , 2004 .

[12]  Beth A Smith,et al.  Determining if wearable sensors affect infant leg movement frequency , 2018, Developmental neurorehabilitation.

[13]  Yann LeCun,et al.  A Closer Look at Spatiotemporal Convolutions for Action Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[14]  Marcin Grzegorzek,et al.  Detection of Infantile Movement Disorders in Video Data Using Deformable Part-Based Model , 2018, Sensors.

[15]  L. Radlinger,et al.  Inter- and intra-observer agreement of Prechtl's method on the qualitative assessment of general movements in preterm, term and young infants. , 2011, Early human development.

[16]  Varun Ramakrishna,et al.  Convolutional Pose Machines , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  M. Hadders‐Algra,et al.  Kinematic Quality of Reaching Movements in Preterm Infants , 2003, Pediatric Research.

[18]  G Cioni,et al.  The qualitative assessment of general movements in preterm, term and young infants--review of the methodology. , 1997, Early human development.

[19]  Lena Maier-Hein,et al.  Uncertainty-Aware Organ Classification for Surgical Data Science Applications in Laparoscopy , 2017, IEEE Transactions on Biomedical Engineering.

[20]  Georgios Tzimiropoulos,et al.  Human Pose Estimation via Convolutional Part Heatmap Regression , 2016, ECCV.

[21]  Jia Deng,et al.  Stacked Hourglass Networks for Human Pose Estimation , 2016, ECCV.

[22]  Sergio Escalera,et al.  Graph cuts optimization for multi-limb human segmentation in depth maps , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[23]  Fay B. Horak,et al.  Daily Quantity of Infant Leg Movement: Wearable Sensor Algorithm and Relationship to Walking Onset , 2015, Sensors.

[24]  Tom Chau,et al.  Detection of Atypical and Typical Infant Movements using Computer-based Video Analysis , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[25]  Danail Stoyanov,et al.  Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks , 2018, IEEE Transactions on Medical Imaging.

[26]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[27]  Emanuele Frontoni,et al.  Non-Contact Monitoring of Preterm Infants Using RGB-D Camera , 2015 .

[28]  Bernt Schiele,et al.  Learning Non-maximum Suppression , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[30]  Y. Schutz,et al.  Energy Balance, Physical Activity, and Thermogenic Effect of Feeding in Premature Infants , 1986, Pediatric Research.

[31]  N. Marlow,et al.  Relationship between test scores using the second and third editions of the Bayley Scales in extremely preterm children. , 2012, The Journal of pediatrics.

[32]  Emanuele Frontoni,et al.  Preterm infants' limb-pose estimation from depth images using convolutional neural networks , 2019, 2019 IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB).

[33]  Elena De Momi,et al.  Long Term Safety Area Tracking (LT‐SAT) with online failure detection and recovery for robotic minimally invasive surgery , 2018, Medical Image Anal..

[34]  P. Cogo,et al.  Increased morbidity and mortality in very preterm/VLBW infants with congenital heart disease , 2013, Intensive Care Medicine.

[35]  K von Siebenthal,et al.  Differences in movement quality at term among preterm and term infants. , 1997, Biology of the neonate.

[36]  G. Greisen,et al.  European Consensus Guidelines on the Management of Respiratory Distress Syndrome - 2016 Update , 2016, Neonatology.