Development and evaluation of a hand tracker using depth images captured from an overhead perspective

We present the development and evaluation of a hand tracking algorithm based on single depth images captured from an overhead perspective for use in the COACH prompting system. We train a random decision forest body part classifier using approximately 5,000 manually labeled, unbalanced, partially labeled training images. The classifier represents a random subset of pixels in each depth image with a learned probability density function across all trained body parts. A local mode-find approach is used to search for clusters present in the underlying feature space sampled by the classified pixels. In each frame, body part positions are chosen as the mode with the highest confidence. User hand positions are translated into hand washing task actions based on proximity to environmental objects. We validate the performance of the classifier and task action proposals on a large set of approximately 24,000 manually labeled images.

[1]  Antonis A. Argyros,et al.  Efficient model-based 3D tracking of hand articulations using Kinect , 2011, BMVC.

[2]  Jesse Hoey,et al.  POMDP Models for Assistive Technology , 2005, AAAI Fall Symposium: Caring Machines.

[3]  Björn W. Schuller,et al.  The INTERSPEECH 2009 emotion challenge , 2009, INTERSPEECH.

[4]  Ming-Lu Wu,et al.  Quality Function Deployment: A Comprehensive Review of Its Concepts and Methods , 2002 .

[5]  Jesse Hoey,et al.  A Decision-Theoretic Approach to Task Assistance for Persons with Dementia , 2005, IJCAI.

[6]  Jesse Hoey,et al.  A planning system based on Markov decision processes to guide people with dementia through activities of daily living , 2006, IEEE Transactions on Information Technology in Biomedicine.

[7]  Alex Mihailidis,et al.  A real-world deployment of the COACH prompting system , 2013, J. Ambient Intell. Smart Environ..

[8]  Pushmeet Kohli,et al.  Key Developments in Human Pose Estimation for Kinect , 2013, Consumer Depth Cameras for Computer Vision.

[9]  Jesse Hoey,et al.  Automated handwashing assistance for persons with dementia using video and a partially observable Markov decision process , 2010, Comput. Vis. Image Underst..

[10]  Richard Bowden,et al.  Static Pose Estimation from Depth Images using Random Regression Forests and Hough Voting , 2012, VISAPP.

[11]  Jesse Hoey,et al.  Assisting persons with dementia during handwashing using a partially observable Markov decision process. , 2007, ICVS 2007.

[12]  J. Barbenel,et al.  The efficacy of an intelligent cognitive orthosis to facilitate handwashing by persons with moderate to severe dementia , 2004 .

[13]  Larry D. Hostetler,et al.  The estimation of the gradient of a density function, with applications in pattern recognition , 1975, IEEE Trans. Inf. Theory.

[14]  Javier Ruiz Hidalgo,et al.  Real-Time Head and Hand Tracking Based on 2.5D Data , 2012 .

[15]  Jesse Hoey.  Tracking using Flocks of Features, with Application to Assisted Handwashing , 2006, BMVC.

[16]  Alex Mihailidis,et al.  The design of intelligent in-home assistive technologies: Assessing the needs of older adults with dementia and their caregivers , 2011 .

[17]  Hocine Cherifi,et al.  Evaluation of Performance Measures for Classifiers Comparison , 2011, UbiComp 2011.

[18]  Sergio Escalera,et al.  Graph cuts optimization for multi-limb human segmentation in depth maps , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Atul Kanaujia,et al.  Part Segmentation of Visual Hull for 3D Human Pose Estimation , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[20]  Reinhard Koch,et al.  Time‐of‐Flight Cameras in Computer Graphics , 2010, Comput. Graph. Forum.

[21]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[22]  Jesse Hoey,et al.  The use of an intelligent prompting system for people with dementia , 2007, Interactions.

[23]  Benjamin J. Southwell,et al.  Human Object Recognition Using Colour and Depth Information from an RGB-D Kinect Sensor , 2013 .

[24]  Alex Mihailidis,et al.  Examining effective communication strategies used by formal caregivers when interacting with Alzheimer’s disease residents during an activity of daily living (ADL) , 2007, Brain and Language.

[25]  A Mihailidis.  The development of an intelligent cognitive orthosis to facilitate handwashing for persons with moderate-to-severe dementia. , 2002 .

[26]  Lale Akarun,et al.  Real time hand pose estimation using depth sensors , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[27]  Alex Mihailidis,et al.  The use of computer vision in an intelligent environment to support aging-in-place, safety, and independence in the home , 2004, IEEE Transactions on Information Technology in Biomedicine.

[28]  R. Venkatesh Babu,et al.  Human action recognition using depth maps , 2012, 2012 International Conference on Signal Processing and Communications (SPCOM).

[29]  A. Mihailidis,et al.  The COACH prompting system to assist older adults with dementia through handwashing: An efficacy study , 2008, BMC geriatrics.

[30]  Susan Carlson Skalak.  House of Quality , 2002 .

[31]  A. Mihailidis,et al.  The use of automated prompting to facilitate handwashing in persons with dementia. , 2006, The American journal of occupational therapy : official publication of the American Occupational Therapy Association.