Long-Term Multi-Cue Tracking of Hands in Vehicles

Hands are a very important cue for understanding and analyzing driver activity and human activity, in general. Vision-based hand detection and tracking involve major challenges, such as attaining robustness to inconsistencies in lighting and scale, background clutter, object occlusion/disappearance and the large variability in hand shape, size, color, and structure. In this paper, we introduce a novel framework suitable for tracking multiple hands online. Assigning tracks to these detections is modeled as a bipartite matching problem with an objective of minimizing the total cost. Both motion and appearance cues are integrated in order to gain robustness to occlusion, fast movement, and interacting hands. Additionally, we study the utility of a left versus right hand classifier to disambiguate hand tracks and reduce ID switches. The proposed tracker shows promise on an extensive, naturalistic, and publicly available driving (VIVA Challenge) data set, by tracking both hands of the driver and the passenger effectively.

[1]  Neil K Chaudhary,et al.  District of Columbia. , 1896, The Journal of comparative medicine and veterinary archives.

[2]  Ramakant Nevatia,et al.  An online learned CRF model for multi-target tracking , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[3]  Thomas B. Moeslund,et al.  Real-time recognition of hand alphabet gestures using principal component analysis , 1997 .

[4]  Martin Lauer,et al.  3D Traffic Scene Understanding From Movable Platforms , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Jiri Matas,et al.  Forward-Backward Error: Automatic Detection of Tracking Failures , 2010, 2010 20th International Conference on Pattern Recognition.

[6]  Eleonora Vig,et al.  Online Domain Adaptation for Multi-Object Tracking , 2015, BMVC.

[7]  Jeremy D Sudweeks,et al.  An Analysis of Driver Inattention Using a Case-Crossover Approach On 100-Car Data: Final Report , 2010 .

[8]  Cordelia Schmid,et al.  Dense Trajectories and Motion Boundary Descriptors for Action Recognition , 2013, International Journal of Computer Vision.

[9]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[10]  Richard J. Hanowski,et al.  Driver Distraction in Commercial Vehicle Operations , 2009 .

[11]  Markus Kohler,et al.  Motion Detection and Tracking under Constraint of Pan-tilt Cameras for Vision-based Human Computer Interaction Motion Detection and Tracking under Constraint of Pan-tilt Cameras for Vision-based Human Computer Interaction , 1998 .

[12]  T. Başar,et al.  A New Approach to Linear Filtering and Prediction Problems , 2001 .

[13]  Ramakant Nevatia,et al.  Global data association for multi-object tracking using network flows , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Rainer Stiefelhagen,et al.  Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics , 2008, EURASIP J. Image Video Process..

[15]  Shan Lu,et al.  Color-based hands tracking system for sign language recognition , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[16]  Zdenek Kalal,et al.  Tracking-Learning-Detection , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Ramakant Nevatia,et al.  Learning to associate: HybridBoosted multi-target tracker for crowded scene , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Tieniu Tan,et al.  Real time hand tracking by combining particle filtering and mean shift , 2004, Sixth IEEE International Conference on Automatic Face and Gesture Recognition, 2004. Proceedings..

[19]  Stan Sclaroff,et al.  Automatic 2D Hand Tracking in Video Sequences , 2005, 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION'05) - Volume 1.

[20]  Konrad Schindler,et al.  Multi-target tracking by continuous energy minimization , 2011, CVPR 2011.

[21]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[22]  Mathias Kölsch,et al.  Fast 2D Hand Tracking with Flocks of Features and Multi-Cue Integration , 2004, 2004 Conference on Computer Vision and Pattern Recognition Workshop.

[23]  Ming-Hsuan Yang,et al.  Bayesian Multi-object Tracking Using Motion Context from Multiple Objects , 2015, 2015 IEEE Winter Conference on Applications of Computer Vision.

[24]  Mohan M. Trivedi,et al.  The Power Is in Your Hands: 3D Analysis of Hand Gestures in Naturalistic Video , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[25]  Mohan M. Trivedi,et al.  Hand Gesture Recognition in Real Time for Automotive Interfaces: A Multimodal Vision-Based Approach and Evaluations , 2014, IEEE Transactions on Intelligent Transportation Systems.

[26]  Mohan M. Trivedi,et al.  Vision for Intelligent Vehicles and Application , 2015 .

[27]  Frank Dellaert,et al.  MCMC-based particle filtering for tracking a variable number of interacting targets , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Manolis I. A. Lourakis,et al.  Real-Time Tracking of Multiple Skin-Colored Objects with a Possibly Moving Camera , 2004, ECCV.

[29]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[30]  Mohan M. Trivedi,et al.  Object detection based on gray level cooccurrence , 1984, Comput. Vis. Graph. Image Process..

[31]  Mohan M. Trivedi,et al.  On Performance Evaluation of Driver Hand Detection Algorithms: Challenges, Dataset, and Metrics , 2015, 2015 IEEE 18th International Conference on Intelligent Transportation Systems.

[32]  Mohan M. Trivedi,et al.  3-D Posture and Gesture Recognition for Interactivity in Smart Spaces , 2012, IEEE Transactions on Industrial Informatics.

[33]  Wongun Choi,et al.  Near-Online Multi-target Tracking with Aggregated Local Flow Descriptor , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[34]  Mohan M. Trivedi,et al.  On surveillance for safety critical events: In-vehicle video networks for predictive driver assistance systems , 2015, Comput. Vis. Image Underst..

[35]  Ian Reid,et al.  Probabilistic tracking and recognition of nonrigid hand motion , 2003, 2003 IEEE International SOI Conference. Proceedings (Cat. No.03CH37443).

[36]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[37]  Mohan M. Trivedi,et al.  Beyond just keeping hands on the wheel: Towards visual interpretation of driver hand motion patterns , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[38]  Paulo R. S. Mendonça,et al.  Model-based 3D tracking of an articulated hand , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[39]  Mohan M. Trivedi,et al.  Learning to Detect Vehicles by Clustering Appearance Patterns , 2015, IEEE Transactions on Intelligent Transportation Systems.

[40]  Charless C. Fowlkes,et al.  Globally-optimal greedy algorithms for tracking a variable number of objects , 2011, CVPR 2011.

[41]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.