Bodies at Rest: 3D Human Pose and Shape Estimation From a Pressure Image Using Synthetic Data

People spend a substantial part of their lives at rest in bed. 3D human pose and shape estimation for this activity would have numerous beneficial applications, yet line-of-sight perception is complicated by occlusion from bedding. Pressure sensing mats are a promising alternative, but training data is challenging to collect at scale. We describe a physics-based method that simulates human bodies at rest in a bed with a pressure sensing mat, and present PressurePose, a synthetic dataset with 206K pressure images with 3D human poses and shapes. We also present PressureNet, a deep learning model that estimates human pose and shape given a pressure image and gender. PressureNet incorporates a pressure map reconstruction (PMR) network that models pressure image generation to promote consistency between estimated 3D body models and pressure image input. In our evaluations, PressureNet performed well with real data from participants in diverse poses, even though it had only been trained with synthetic data. When we ablated the PMR network, performance dropped substantially.

[1]  Mehrdad Nourani,et al.  Detecting high-risk regions for pressure ulcer risk assessment , 2013, 2013 IEEE International Conference on Bioinformatics and Biomedicine.

[2]  Ioannis A. Kakadiaris,et al.  3D Human pose estimation: A review of the literature and analysis of covariates , 2016, Comput. Vis. Image Underst..

[3]  Christian Szegedy,et al.  DeepPose: Human Pose Estimation via Deep Neural Networks , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[4]  Tae-Yong Kim,et al.  Unified particle physics for real-time applications , 2014, ACM Trans. Graph..

[5]  Wei Zhang,et al.  Deep Kinematic Pose Regression , 2016, ECCV Workshops.

[6]  Charles C. Kemp,et al.  ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[7]  Gregory D. Hager,et al.  Characterization and simulation of tactile sensors , 2010, 2010 IEEE Haptics Symposium.

[8]  Lucia Melloni,et al.  Patient-Specific Pose Estimation in Clinical Environments , 2018, IEEE Journal of Translational Engineering in Health and Medicine.

[9]  Cordelia Schmid,et al.  Learning from Synthetic Humans , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[11]  Jitendra Malik,et al.  End-to-End Recovery of Human Shape and Pose , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[12]  Peter V. Gehler,et al.  Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image , 2016, ECCV.

[13]  C. Karen Liu,et al.  Deep Haptic Model Predictive Control for Robot-Assisted Dressing , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[14]  Dimitrios Tzionas,et al.  Expressive Body Capture: 3D Hands, Face, and Body From a Single Image , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  S. Azen,et al.  Normal range of motion of joints in male subjects. , 1979, The Journal of bone and joint surgery. American volume.

[16]  Majid Sarrafzadeh,et al.  Bodypart localization for pressure ulcer prevention , 2014, 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[17]  Ahsan Habib,et al.  SkinSim: A simulation environment for multimodal robot skin , 2014, 2014 IEEE International Conference on Automation Science and Engineering (CASE).

[18]  Michael J. Black,et al.  SMPL: A Skinned Multi-Person Linear Model , 2023 .

[19]  Michael J. Black,et al.  Learning Human Optical Flow , 2018, BMVC.

[20]  Georgios Tzimiropoulos,et al.  Human Pose Estimation via Convolutional Part Heatmap Regression , 2016, ECCV.

[21]  Nassir Navab,et al.  Patient 3D body pose estimation from pressure imaging , 2018, International Journal of Computer Assisted Radiology and Surgery.

[22]  AgarwalAnkur,et al.  Recovering 3D Human Pose from Monocular Images , 2006 .

[23]  Sarah Ostadabbas,et al.  Seeing Under the Cover: A Physics Guided Learning Approach for In-Bed Pose Estimation , 2019, MICCAI.

[24]  Sebastian Thrun,et al.  SCAPE: shape completion and animation of people , 2005, SIGGRAPH '05.

[25]  Charles C. Kemp,et al.  3D Human Pose Estimation on a Configurable Bed from a Pressure Image , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[26]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[27]  Tomomasa Sato,et al.  Body parts positions and posture estimation system based on pressure distribution image , 1999, Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C).

[28]  Xiaowei Zhou,et al.  Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Stefano Soatto,et al.  Relevant Feature Selection for Human Pose Estimation and Localization in Cluttered Images , 2008, ECCV.

[30]  Jonathan Tompson,et al.  Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation , 2014, NIPS.

[31]  Jia Deng,et al.  Stacked Hourglass Networks for Human Pose Estimation , 2016, ECCV.

[32]  Tomomasa Sato,et al.  Pressure distribution image based human motion tracking system using skeleton and surface integration model , 2001, Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164).

[33]  Siddhartha S. Srinivasa,et al.  DART: Dynamic Animation and Robotics Toolkit , 2018, J. Open Source Softw..

[34]  Nasser Kehtarnavaz,et al.  In-bed posture classification and limb identification , 2014, 2014 IEEE Biomedical Circuits and Systems Conference (BioCAS) Proceedings.

[35]  Günther Greiner,et al.  Markerless estimation of patient orientation, posture and pose using range and pressure imaging , 2012, International Journal of Computer Assisted Radiology and Surgery.

[36]  Shahram Payandeh,et al.  Finite elements, mass-spring-damper systems and haptic rendering , 2001, Proceedings 2001 IEEE International Symposium on Computational Intelligence in Robotics and Automation (Cat. No.01EX515).

[37]  Ian Millington.  Game Physics Engine Development: How to Build a Robust Commercial-Grade Physics Engine for your Game , 2010 .

[38]  Greg Turk,et al.  Learning to navigate cloth using haptics , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[39]  Petros Koumoutsakos,et al.  Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) , 2003, Evolutionary Computation.

[40]  Dimitrios Tzionas,et al.  Resolving 3D Human Pose Ambiguities With 3D Scene Constraints , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[41]  Peter Kuster,et al.  Human Body Dynamics Classical Mechanics And Human Movement , 2016 .

[42]  Qionghai Dai,et al.  SimulCap : Single-View Human Performance Capture With Cloth Simulation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[43]  G. Andersson,et al.  Normal range of motion of the hip, knee and ankle joints in male subjects, 30-40 years of age. , 1982, Acta orthopaedica Scandinavica.

[44]  Michael J. Black,et al.  Pose-conditioned joint angle limits for 3D human pose reconstruction , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  C. Karen Liu,et al.  Assistive Gym: A Physics Simulation Framework for Assistive Robotics , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[46]  Sarah Ostadabbas,et al.  In-Bed Pose Estimation: Deep Learning With Shallow Dataset , 2017, IEEE Journal of Translational Engineering in Health and Medicine.

[47]  Ankur Agarwal,et al.  Recovering 3D human pose from monocular images , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[48]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[49]  Ali Etemad,et al.  In-Bed Pressure-Based Pose Estimation Using Image Space Representation Learning , 2019, ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[50]  Jeffrey C. Trinkle,et al.  BubbleTouch: A Quasi-Static Tactile Skin Simulator , 2018, ArXiv.

[51]  Cordelia Schmid,et al.  Learning Joint Reconstruction of Hands and Manipulated Objects , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Nassir Navab,et al.  Patient MoCap: Human Pose Estimation Under Blanket Occlusion for Hospital Monitoring Applications , 2016, MICCAI.

[53]  Zhenhua Wang,et al.  Synthesizing Training Images for Boosting Human 3D Pose Estimation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[54]  Jitendra Malik,et al.  Human Pose Estimation with Iterative Error Feedback , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Michael J. Black,et al.  Resolving 3 D Human Pose Ambiguities with 3 D Scene Constraints , 2019 .

[56]  J. Soucie,et al.  Range of motion measurements: reference values and a database for comparison studies , 2011, Haemophilia : the official journal of the World Federation of Hemophilia.