A tutorial on human activity recognition using body-worn inertial sensors

The last 20 years have seen ever-increasing research activity in the field of human activity recognition. With activity recognition having considerably matured, so has the number of challenges in designing, implementing, and evaluating activity recognition systems. This tutorial aims to provide a comprehensive hands-on introduction for newcomers to the field of human activity recognition. It specifically focuses on activity recognition using on-body inertial sensors. We first discuss the key research challenges that human activity recognition shares with general pattern recognition and identify those challenges that are specific to human activity recognition. We then describe the concept of an Activity Recognition Chain (ARC) as a general-purpose framework for designing and evaluating activity recognition systems. We detail each component of the framework, provide references to related research, and introduce the best practice methods developed by the activity recognition research community. We conclude with the educational example problem of recognizing different hand gestures from inertial sensors attached to the upper and lower arm. We illustrate how each component of this framework can be implemented for this specific activity recognition problem and demonstrate how different implementations compare and how they impact overall recognition performance.

[1]  Paul Lukowicz,et al.  AMON: a wearable multiparameter medical monitoring and alert system , 2004, IEEE Transactions on Information Technology in Biomedicine.

[2]  Mi Zhang,et al.  Motion primitive-based human activity recognition using a bag-of-features approach , 2012, IHI '12.

[3]  Pavel Pudil,et al.  Flexible-Hybrid Sequential Floating Search in Statistical Feature Selection , 2006, SSPR/SPR.

[4]  J. Friedman Special Invited Paper-Additive logistic regression: A statistical view of boosting , 2000 .

[5]  Paul Lukowicz,et al.  Active Capacitive Sensing: Exploring a New Wearable Sensing Modality for Activity Recognition , 2010, Pervasive.

[6]  Alex Pentland,et al.  Unsupervised clustering of ambulatory audio and video , 1999, 1999 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258).

[7]  David R Bassett,et al.  2011 Compendium of Physical Activities: a second update of codes and MET values. , 2011, Medicine and science in sports and exercise.

[8]  Bernt Schiele,et al.  Multi Activity Recognition Based on Bodymodel-Derived Primitives , 2009, LoCA.

[9]  Wei Pan,et al.  SoundSense: scalable sound sensing for people-centric applications on mobile phones , 2009, MobiSys '09.

[10]  Bernt Schiele,et al.  Sensing Location in Your Pocket , 2008, UbiComp 2008.

[11]  Joseph A. Paradiso,et al.  Shoe-integrated sensor system for wireless gait analysis and real-time feedback , 2002, Proceedings of the Second Joint 24th Annual Conference and the Annual Fall Meeting of the Biomedical Engineering Society] [Engineering in Medicine and Biology.

[12]  G. Englebienne,et al.  Transferring Knowledge of Activity Recognition across Sensor Networks , 2010, Pervasive.

[13]  Fuhui Long,et al.  Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy , 2003, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  S. Katz,et al.  Progress in development of the index of ADL. , 1970, The Gerontologist.

[15]  Bernt Schiele,et al.  Scalable Recognition of Daily Activities with Wearable Sensors , 2007, LoCA.

[16]  Zhigang Liu,et al.  The Jigsaw continuous sensing engine for mobile phone applications , 2010, SenSys '10.

[17]  Paul Lukowicz,et al.  All for one or one for all? Combining heterogeneous features for activity spotting , 2010, 2010 8th IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops).

[18]  Gaetano Borriello,et al.  A Practical Approach to Recognizing Physical Activities , 2006, Pervasive.

[19]  Marco Luca Sbodio,et al.  A Wearable Computing Prototype for supporting training activities in Automotive Production , 2007 .

[20]  Rama Chellappa,et al.  Machine Recognition of Human Activities: A Survey , 2008, IEEE Transactions on Circuits and Systems for Video Technology.

[21]  Yangsheng Xu,et al.  Online, interactive learning of gestures for human/robot interfaces , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[22]  Hassan Ghasemzadeh,et al.  An automatic segmentation technique in body sensor networks based on signal energy , 2009, BODYNETS.

[23]  Bernt Schiele,et al.  Remember and transfer what you have learned - recognizing composite activities based on activity spotting , 2010, International Symposium on Wearable Computers (ISWC) 2010.

[24]  Thad Starner,et al.  MAGIC: a motion gesture design tool , 2010, CHI.

[25]  Franz Gravenhorst,et al.  An IMU-based sensor network to continuously monitor rowing technique on the water , 2011, 2011 Seventh International Conference on Intelligent Sensors, Sensor Networks and Information Processing.

[26]  Gerhard Tröster,et al.  Recognizing Upper Body Postures using Textile Strain Sensors , 2007, 2007 11th IEEE International Symposium on Wearable Computers.

[27]  Paul Lukowicz,et al.  Activity Recognition of Assembly Tasks Using Body-Worn Microphones and Accelerometers , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[28]  Jianguo Zhang,et al.  The PASCAL Visual Object Classes Challenge , 2006 .

[29]  Aaron F. Bobick,et al.  Realtime online adaptive gesture recognition , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[30]  Antonio Torralba,et al.  Sharing Visual Features for Multiclass and Multiview Object Detection , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[31]  Thad Starner,et al.  Using GPS to learn significant locations and predict movement across multiple users , 2003, Personal and Ubiquitous Computing.

[32]  Moisés Goldszmidt,et al.  Properties and Benefits of Calibrated Classifiers , 2004, PKDD.

[33]  Cassim Ladha,et al.  ClimbAX: skill assessment for climbing enthusiasts , 2013, UbiComp.

[34]  Gerhard Tröster,et al.  Gestures are strings: efficient online gesture spotting and classification using string matching , 2007, BODYNETS.

[35]  Gerhard Tröster,et al.  Recognition of Hearing Needs from Body and Eye Movements to Improve Hearing Instruments , 2011, Pervasive.

[36]  Agnes Grünerbl,et al.  The benefit of activity recognition for mobile phone based nursing documentation: A Wizard-of-Oz study , 2010, International Symposium on Wearable Computers (ISWC) 2010.

[37]  David Wetherall,et al.  Recognizing daily activities with RFID-based sensors , 2009, UbiComp.

[38]  Kristof Van Laerhoven,et al.  When Else Did This Happen? Efficient Subsequence Representation and Matching for Wearable Activity Data , 2009, 2009 International Symposium on Wearable Computers.

[39]  Darren Leigh,et al.  SocialMotion: Measuring the Hidden Social Life of a Building , 2007, LoCA.

[40]  Kristof Van Laerhoven,et al.  What shall we teach our pants? , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[41]  Eric Horvitz,et al.  Experience sampling for building predictive user models: a comparative study , 2008, CHI.

[42]  Bernt Schiele,et al.  Using rhythm awareness in long-term activity recognition , 2008, 2008 12th IEEE International Symposium on Wearable Computers.

[43]  R. Bajcsy,et al.  Wearable Sensors for Reliable Fall Detection , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[44]  Patrick Olivier,et al.  The mobile fitness coach: Towards individualized skill assessment using personalized mobile devices , 2013, Pervasive Mob. Comput..

[45]  Ulf Blanke Sensing Location in the Pocket , 2008 .

[46]  Paul Lukowicz,et al.  Performance metrics for activity recognition , 2011, TIST.

[47]  Gregory D. Abowd,et al.  Recognizing mimicked autistic self-stimulatory behaviors using HMMs , 2005, Ninth IEEE International Symposium on Wearable Computers (ISWC'05).

[48]  Albrecht Schmidt,et al.  Multi-sensor context aware clothing , 2002, Proceedings. Sixth International Symposium on Wearable Computers,.

[49]  Hugo Fuks,et al.  Qualitative activity recognition of weight lifting exercises , 2013, AH.

[50]  Daniel Roggen,et al.  Recognition of visual memory recall processes using eye movement analysis , 2011, UbiComp '11.

[51]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[52]  Michael L. Littman,et al.  Activity Recognition from Accelerometer Data , 2005, AAAI.

[53]  Alex Pentland,et al.  Wearable feedback systems for rehabilitation , 2005, Journal of NeuroEngineering and Rehabilitation.

[54]  Eric Horvitz,et al.  Predestination: Inferring Destinations from Partial Trajectories , 2006, UbiComp.

[55]  Bernt Schiele,et al.  Analyzing features for activity recognition , 2005, sOc-EUSAI '05.

[56]  Alex Pentland,et al.  Healthwear: medical technology becomes wearable , 2004, Computer.

[57]  Sargur N. Srihari,et al.  Decision Combination in Multiple Classifier Systems , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[58]  Paul Lukowicz,et al.  Where am I: Recognizing On-body Positions of Wearable Sensors , 2005, LoCA.

[59]  Alex Pentland,et al.  A wearable computer-based American sign Language Recogniser , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[60]  Paul Lukowicz,et al.  Wearable Activity Tracking in Car Manufacturing , 2008, IEEE Pervasive Computing.

[61]  Gernot A. Fink,et al.  Markov Models for Pattern Recognition: From Theory to Applications , 2007 .

[62]  Gerhard Tröster,et al.  Probabilistic parsing of dietary activity events , 2007, BSN.

[63]  Jeffrey M. Hausdorff,et al.  Potentials of Enhanced Context Awareness in Wearable Assistants for Parkinson's Disease Patients with the Freezing of Gait Syndrome , 2009, 2009 International Symposium on Wearable Computers.

[64]  Diogo R. Ferreira,et al.  Preprocessing techniques for context recognition from accelerometer data , 2010, Personal and Ubiquitous Computing.

[65]  Paul Lukowicz,et al.  Towards Recognizing Tai Chi - An Initial Experiment Using Wearable Sensors , 2006 .

[66]  R. Polikar,et al.  Ensemble based systems in decision making , 2006, IEEE Circuits and Systems Magazine.

[67]  Alex Pentland,et al.  A Wearable Computer Based American Sign Language Recognizer , 1998, Assistive Technology and Artificial Intelligence.

[68]  Dadong Wan,et al.  Magic Medicine Cabinet: A Situated Portal for Consumer Healthcare , 1999, HUC.

[69]  Gregory D. Abowd,et al.  Context-awareness in wearable and ubiquitous computing , 1997, Digest of Papers. First International Symposium on Wearable Computers.

[70]  Gerhard Tröster,et al.  Detection of eating and drinking arm gestures using inertial body-worn sensors , 2005, Ninth IEEE International Symposium on Wearable Computers (ISWC'05).

[71]  Bernt Schiele,et al.  An Analysis of Sensor-Oriented vs. Model-Based Activity Recognition , 2009, 2009 International Symposium on Wearable Computers.

[72]  Ricardo Chavarriaga,et al.  Unsupervised Adaptation to On-body Sensor Displacement in Acceleration-Based Activity Recognition , 2011, 2011 15th Annual International Symposium on Wearable Computers.

[73]  Oliver Amft Self-Taught Learning for Activity Spotting in On-body Motion Sensor Data , 2011, 2011 15th Annual International Symposium on Wearable Computers.

[74]  Jiri Matas,et al.  On Combining Classifiers , 1998, IEEE Trans. Pattern Anal. Mach. Intell..

[75]  Isabelle Guyon,et al.  An Introduction to Variable and Feature Selection , 2003, J. Mach. Learn. Res..

[76]  Nuria Oliver,et al.  MoviPill: improving medication compliance for elders using a mobile persuasive social game , 2010, UbiComp.

[77]  Hans-Werner Gellersen,et al.  Multimodal recognition of reading activity in transit using body-worn sensors , 2012, TAP.

[78]  Grant Schindler,et al.  A Wearable Interface for Topological Mapping and Localization in Indoor Environments , 2006, LoCA.

[79]  Henry A. Kautz,et al.  Fine-grained activity recognition by aggregating abstract object usage , 2005, Ninth IEEE International Symposium on Wearable Computers (ISWC'05).

[80]  Hans-Werner Gellersen,et al.  EyeContext: recognition of high-level contextual cues from human visual behaviour , 2013, CHI.

[81]  Kent Larson,et al.  Activity Recognition in the Home Using Simple and Ubiquitous Sensors , 2004, Pervasive.

[82]  Blake Hannaford,et al.  A Hybrid Discriminative/Generative Approach for Modeling Human Activities , 2005, IJCAI.

[83]  Patrick Olivier,et al.  Feature Learning for Activity Recognition in Ubiquitous Computing , 2011, IJCAI.

[84]  Alex Pentland,et al.  Recognizing user context via wearable sensors , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[85]  Bharti Bansal,et al.  Gesture Recognition: A Survey , 2016 .

[86]  T. Kuo,et al.  The application of cepstral coefficients and maximum likelihood method in EMG pattern recognition. , 1995, IEEE transactions on bio-medical engineering.

[87]  Bernt Schiele,et al.  Daily Routine Recognition through Activity Spotting , 2009, LoCA.

[88]  Ron Kohavi,et al.  Wrappers for Feature Subset Selection , 1997, Artif. Intell..

[89]  Agata Rozga,et al.  Automatic assessment of problem behavior in individuals with developmental disabilities , 2012, UbiComp.

[90]  Henry A. Kautz,et al.  Inferring activities from interactions with objects , 2004, IEEE Pervasive Computing.

[91]  Donald J. Patterson,et al.  Involuntary gesture recognition for predicting cerebral palsy in high-risk infants , 2010, International Symposium on Wearable Computers (ISWC) 2010.

[92]  共立出版株式会社 コンピュータ・サイエンス : ACM computing surveys , 1978 .

[93]  Lars Widmer,et al.  An Educational and Research Kit for Activity and Context Recognition from On-body Sensors , 2010, 2010 International Conference on Body Sensor Networks.

[94]  Henry A. Kautz,et al.  Location-Based Activity Recognition using Relational Markov Networks , 2005, IJCAI.

[95]  Matthai Philipose,et al.  Common Sense Based Joint Training of Human Activity Recognizers , 2007, IJCAI.

[96]  Bernt Schiele,et al.  ADL recognition based on the combination of RFID and accelerometer sensing , 2008, 2008 Second International Conference on Pervasive Computing Technologies for Healthcare.

[97]  Gerhard Tröster,et al.  The adARC pattern analysis architecture for adaptive human activity recognition systems , 2011, Journal of Ambient Intelligence and Humanized Computing.

[98]  Gwenn Englebienne,et al.  Accurate activity recognition in a home setting , 2008, UbiComp.

[99]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[100]  Nuria Oliver,et al.  HealthGear: Automatic Sleep Apnea Detection and Monitoring with a Mobile Phone , 2007, J. Commun..

[101]  Bernt Schiele,et al.  Discovery of activity patterns using topic models , 2008 .

[102]  Paul Lukowicz,et al.  Dealing with sensor displacement in motion-based onbody activity recognition systems , 2008, UbiComp.

[103]  Qiang Yang,et al.  Cross-domain activity recognition , 2009, UbiComp.

[104]  Hans-Werner Gellersen,et al.  MotionMA: motion modelling and analysis by demonstration , 2013, CHI.

[105]  Irfan A. Essa,et al.  Discovering Characteristic Actions from On-Body Sensor Data , 2006, 2006 10th IEEE International Symposium on Wearable Computers.

[106]  C. Randell,et al.  Context awareness by analysing accelerometer data , 2000, Digest of Papers. Fourth International Symposium on Wearable Computers.

[107]  Bart Selman,et al.  Human Activity Detection from RGBD Images , 2011, Plan, Activity, and Intent Recognition.

[108]  Gerhard Tröster,et al.  Eye Movement Analysis for Activity Recognition Using Electrooculography , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[109]  Bernt Schiele,et al.  Weakly Supervised Recognition of Daily Life Activities with Wearable Sensors , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[110]  Paul Lukowicz,et al.  Adapting magnetic resonant coupling based relative positioning technology for wearable activitiy recogniton , 2008, 2008 12th IEEE International Symposium on Wearable Computers.

[111]  Paul Lukowicz,et al.  Detecting and Interpreting Muscle Activity with Wearable Force Sensors , 2006, Pervasive.

[112]  Paul Lukowicz,et al.  Performance Metrics and Evaluation Issues for Continuous Activity Recognition , 2006 .

[113]  Ling Bao,et al.  Activity Recognition from User-Annotated Acceleration Data , 2004, Pervasive.

[114]  Paul Lukowicz,et al.  OPPORTUNITY: Towards opportunistic activity and context recognition systems , 2009, 2009 IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks & Workshops.

[115]  Bernt Schiele,et al.  South by South-East or Sitting at the Desk: Can Orientation be a Place? , 2011, 2011 15th Annual International Symposium on Wearable Computers.

[116]  Gerhard Tröster,et al.  Robust Recognition of Reading Activity in Transit Using Wearable Electrooculography , 2009, Pervasive.

[117]  J.K. Aggarwal,et al.  Human activity analysis , 2011, ACM Comput. Surv..

[118]  BullingAndreas,et al.  A tutorial on human activity recognition using body-worn inertial sensors , 2014 .

[119]  Philippe Golle,et al.  On using existing time-use study data for ubiquitous computing applications , 2008, UbiComp.

[120]  Paul Lukowicz,et al.  Using a complex multi-modal on-body sensor system for activity spotting , 2008, 2008 12th IEEE International Symposium on Wearable Computers.

[121]  Masatoshi Ishikawa,et al.  Augmenting spatial awareness with Haptic Radar , 2006, 2006 10th IEEE International Symposium on Wearable Computers.

[122]  L. Benini,et al.  Activity recognition from on-body sensors by classifier fusion: sensor scalability and robustness , 2007, 2007 3rd International Conference on Intelligent Sensors, Sensor Networks and Information.

[123]  Kenji Mase,et al.  Activity and Location Recognition Using Wearable Sensors , 2002, IEEE Pervasive Comput..

[124]  ZissermanAndrew,et al.  The Pascal Visual Object Classes Challenge , 2015 .

[125]  Jennifer Healey,et al.  A Long-Term Evaluation of Sensing Modalities for Activity Recognition , 2007, UbiComp.

[126]  Gerhard Tröster,et al.  Eye Movement Analysis for Activity Recognition Using Electrooculography , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[127]  Scott R. Klemmer,et al.  Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition , 2007, CHI.

[128]  M. Everingham The PASCAL Visual Object Classes Challenge 2005 Development Kit , 2005 .

[129]  Daniel P. Siewiorek,et al.  Activity recognition and monitoring using multiple sensors on different body positions , 2006, International Workshop on Wearable and Implantable Body Sensor Networks (BSN'06).

[130]  Alex Pentland,et al.  Recognizing User's Context from Wearable Sensors: Baseline System , 2000 .

[131]  G. ÓLaighin,et al.  Direct measurement of human movement by accelerometry. , 2008, Medical engineering & physics.

[132]  Vigneshwaran Subbaraju,et al.  Energy-Efficient Continuous Activity Recognition on Mobile Phones: An Activity-Adaptive Approach , 2012, 2012 16th International Symposium on Wearable Computers.