Combining Motion Sensors and Ultrasonic Hands Tracking for Continuous Activity Recognition in a Maintenance Scenario

We present a novel method for continuous activity recognition based on ultrasonic hand tracking and motion sensors attached to the user's arms. It builds on previous work in which we have shown such a sensor combination to be effective for isolated recognition in manually segmented data. We describe the hand tracking based segmentation, show how classification is done on both the ultrasonic and the motion data and discuss different classifier fusion methods. The performance of our method is investigated in a large scale experiment in which typical bicycle repair actions are performed by 6 different subjects. The experiment contains a test set with 1008 activities from 21 classes encompassing 115 minutes randomly mixed with 252 minutes of 'NULL' class. To come as close as possible to a real life continuous scenario we have ensured a diverse and complex 'NULL' class, diverse and often similar activities, inter person training/testing and an additional data set only for training (299 extra minutes of data). A key result of the paper is that our method can handle the user independent testing (testing on users that were not seen in training) nearly as well as the user dependent case.

[1]  Henry A. Kautz,et al.  Fine-grained activity recognition by aggregating abstract object usage , 2005, Ninth IEEE International Symposium on Wearable Computers (ISWC'05).

[2]  Ling Bao,et al.  Activity Recognition from User-Annotated Acceleration Data , 2004, Pervasive.

[3]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[4]  Paul Lukowicz,et al.  Using ultrasonic hand tracking to augment motion analysis based recognition of manipulative gestures , 2005, Ninth IEEE International Symposium on Wearable Computers (ISWC'05).

[5]  Yangsheng Xu,et al.  Online, interactive learning of gestures for human/robot interfaces , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[6]  Dimitris N. Metaxas,et al.  ASL recognition based on a coupling between HMMs and 3D motion analysis , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[7]  Henk L. Muller,et al.  Particle Filters for Position Sensing with Asynchronous Ultrasonic Beacons , 2006, LoCA.

[8]  Alex Pentland,et al.  Visual contextual awareness in wearable computing , 1998, Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215).

[9]  Paul Lukowicz,et al.  Evaluating Performance in Continuous Context Recognition Using Event-Driven Error Characterisation , 2006, LoCA.

[10]  Paul Lukowicz,et al.  Continuous recognition of arm activities with body-worn inertial sensors , 2004, Eighth International Symposium on Wearable Computers.

[11]  C. Randell,et al.  Context awareness by analysing accelerometer data , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[12]  Paul Lukowicz,et al.  SoundButton: design of a low power wearable audio classification system , 2003, Seventh IEEE International Symposium on Wearable Computers, 2003. Proceedings..

[13]  Hari Balakrishnan,et al.  Tracking moving devices with the cricket location system , 2004, MobiSys '04.

[14]  Florian Michahelles,et al.  Proactive Instructions for Furniture Assembly , 2002, UbiComp.

[15]  Paul Lukowicz,et al.  Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Gerd Kortuem,et al.  A relative positioning system for co-located mobile devices , 2005, MobiSys '05.

[17]  William C. Mann,et al.  Enabling location-aware pervasive computing applications for the elderly , 2003, Proceedings of the First IEEE International Conference on Pervasive Computing and Communications, 2003. (PerCom 2003)..

[18]  Paul Lukowicz,et al.  Recognizing Workshop Activity Using Body Worn Microphones and Accelerometers , 2004, Pervasive.

[19]  Junji Yamato,et al.  Recognizing human action in time-sequential images using hidden Markov model , 1992, Proceedings 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[20]  Kenji Mase,et al.  Activity and Location Recognition Using Wearable Sensors , 2002, IEEE Pervasive Comput..

[21]  Jiangwen Deng,et al.  An HMM-based approach for gesture segmentation and recognition , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.