Assessing Eating Behaviour Using Upper Limb Mounted Motion Sensors: A Systematic Review

Wearable motion tracking sensors are now widely used to monitor physical activity, and have recently gained more attention in dietary monitoring research. The aim of this review is to synthesise research to date that utilises upper limb motion tracking sensors, either individually or in combination with other technologies (e.g., cameras, microphones), to objectively assess eating behaviour. Eleven electronic databases were searched in January 2019, and 653 distinct records were obtained. Including 10 studies found in backward and forward searches, a total of 69 studies met the inclusion criteria, with 28 published since 2017. Fifty studies were conducted exclusively in laboratory settings, 13 exclusively in free-living settings, and three in both settings. The most commonly used motion sensor was an accelerometer (64) worn on the wrist (60) or lower arm (5), while in most studies (45), accelerometers were used in combination with gyroscopes. Twenty-six studies used commercial-grade smartwatches or fitness bands, 11 used professional grade devices, and 32 used standalone sensor chipsets. The most used machine learning approaches were Support Vector Machine (SVM, n = 21), Random Forest (n = 19), Decision Tree (n = 16), Hidden Markov Model (HMM, n = 10) algorithms, and from 2017 Deep Learning (n = 5). While comparisons of the detection models are not valid due to the use of different datasets, the models that consider the sequential context of data across time, such as HMM and Deep Learning, show promising results for eating activity detection. We discuss opportunities for future research and emerging applications in the context of dietary assessment and monitoring.

[1]  Konstantinos Kyritsis,et al.  Food Intake Detection from Inertial Sensors Using LSTM Networks , 2017, ICIAP Workshops.

[2]  Andreas Savvides,et al.  Recognizing activities from context and arm pose using finite state machines , 2009, 2009 Third ACM/IEEE International Conference on Distributed Smart Cameras (ICDSC).

[3]  John Paul Varkey,et al.  Human motion recognition using a wireless sensor-based wearable system , 2012, Personal and Ubiquitous Computing.

[4]  Raul I. Ramos-Garcia,et al.  A Study of Temporal Action Sequencing During Consumption of a Meal , 2013, BCB.

[5]  Guanling Chen,et al.  Automatic Eating Detection using head-mount and wrist-worn accelerometers , 2015, 2015 17th International Conference on E-health Networking, Application & Services (HealthCom).

[6]  C. J. Boushey,et al.  New mobile methods for dietary assessment: review of image-assisted and image-based dietary assessment methods , 2016, Proceedings of the Nutrition Society.

[7]  Sergio Escalera,et al.  Action detection fusing multiple Kinects and a WIMU: an application to in-home assistive technology for the elderly , 2018, Machine Vision and Applications.

[8]  Bonnie Spring,et al.  Food watch: detecting and characterizing eating episodes through feeding gestures , 2016 .

[9]  David R Bassett,et al.  Validity of the Bite Counter device in a controlled laboratory setting. , 2014, Eating behaviors.

[10]  F. Delpeuch,et al.  Measuring dietary diversity in rural Burkina Faso: comparison of a 1-day and a 3-day dietary recall , 2007, Public Health Nutrition.

[11]  Nabil Alshurafa,et al.  I sense overeating: Motif-based machine learning framework to detect overeating using wrist-worn sensing , 2018, Inf. Fusion.

[12]  Guoliang Xing,et al.  FamilyLog: A mobile system for monitoring family mealtime activities , 2017, 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom).

[13]  Maysam Ghovanloo,et al.  Unobtrusive and Wearable Systems for Automatic Dietary Monitoring , 2017, IEEE Transactions on Biomedical Engineering.

[14]  Seng Wai Loke,et al.  Assigning Group Activity Semantics to Multi-Device Mobile Sensor Data , 2017, KI - Künstliche Intelligenz.

[15]  Ahyoung Choi,et al.  Asian-style food intake pattern estimation based on convolutional neural network , 2018, 2018 IEEE International Conference on Consumer Electronics (ICCE).

[16]  Subir Biswas,et al.  Meal-time and duration monitoring using wearable sensors , 2017, Biomed. Signal Process. Control..

[17]  Ramón F. Brena,et al.  Multi-view stacking for activity recognition with sound and accelerometer data , 2018, Inf. Fusion.

[18]  M.H. Ang,et al.  Detection of activities for daily life surveillance: Eating and drinking , 2008, HealthCom 2008 - 10th International Conference on e-health Networking, Applications and Services.

[19]  Paul Johns,et al.  Predicting "About-to-Eat" Moments for Just-in-Time Eating Intervention , 2016, Digital Health.

[20]  Statistical Annex , 1995 .

[21]  Gerhard Tröster,et al.  Detection of eating and drinking arm gestures using inertial body-worn sensors , 2005, Ninth IEEE International Symposium on Wearable Computers (ISWC'05).

[22]  E. Opisso,et al.  Monitoring of upper-limb movements through inertial sensors – Preliminary results , 2019, Smart Health.

[23]  Yujie Dong,et al.  Detecting Periods of Eating During Free-Living by Tracking Wrist Motion , 2014, IEEE Journal of Biomedical and Health Informatics.

[24]  Paul Lukowicz,et al.  Towards wearable sensing-based assessment of fluid intake , 2010, 2010 8th IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops).

[25]  Ahmad Lotfi,et al.  A Hierarchical Approach in Food and Drink Intake Recognition Using Wearable Inertial Sensors , 2018, PETRA.

[26]  Paul J. M. Havinga,et al.  A hierarchical lazy smoking detection algorithm using smartwatch sensors , 2016, 2016 IEEE 18th International Conference on e-Health Networking, Applications and Services (Healthcom).

[27]  Adam W. Hoover,et al.  A New Method for Measuring Meal Intake in Humans via Automated Wrist Motion Tracking , 2012, Applied Psychophysiology and Biofeedback.

[28]  Paolo Dario,et al.  Daily activity recognition with inertial ring and bracelet: An unsupervised approach , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[29]  Gregory D. Abowd,et al.  A practical approach for recognizing eating moments with wrist-mounted inertial sensing , 2015, UbiComp.

[30]  Adam W. Hoover,et al.  Byte by Bite: Use of a mobile Bite Counter and weekly behavioral challenges to promote weight loss. , 2017, Smart health.

[31]  P. Ciampolini,et al.  Automatic diet monitoring: a review of computer vision and wearable sensor-based methods , 2017, International journal of food sciences and nutrition.

[32]  Edward Sazonov,et al.  A Systematic Review of Technology-Driven Methodologies for Estimation of Energy Intake , 2019, IEEE Access.

[33]  Chen-Khong Tham,et al.  Eating activity primitives detection - a step towards ADL recognition , 2008, HealthCom 2008 - 10th International Conference on e-health Networking, Applications and Services.

[34]  Bernt Schiele,et al.  Using rhythm awareness in long-term activity recognition , 2008, 2008 12th IEEE International Symposium on Wearable Computers.

[35]  Eric R Muth,et al.  Slowing bite-rate reduces energy intake: an application of the bite counter device. , 2011, Journal of the American Dietetic Association.

[36]  Tracy Burrows,et al.  Dietary Assessment of Shared Plate Eating: A Missing Link , 2019, Nutrients.

[37]  Edward Sazonov,et al.  Automatic Ingestion Monitor: A Novel Wearable Device for Monitoring of Ingestive Behavior , 2014, IEEE Transactions on Biomedical Engineering.

[38]  Sergio Parra-Sánchez,et al.  Recognition of activities of daily living based on the vertical displacement of the wrist , 2016 .

[39]  Joohee Kim,et al.  An eating speed guide system using a wristband and tabletop unit , 2016, UbiComp Adjunct.

[40]  Adam W. Hoover,et al.  The Impact of Quantity of Training Data on Recognition of Eating Gestures , 2018, ArXiv.

[41]  Edward Sazonov,et al.  Estimation of feature importance for food intake detection based on Random Forests classification , 2013, 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[42]  Hyun-Jun Kim,et al.  Eating activity recognition for health and wellness: A case study on Asian eating style , 2013, 2013 IEEE International Conference on Consumer Electronics (ICCE).

[43]  Megan E. Rollo,et al.  Evaluation of a Mobile Phone Image-Based Dietary Assessment Method in Adults with Type 2 Diabetes , 2015, Nutrients.

[44]  Edward Sazonov,et al.  Detection of chewing from piezoelectric film sensor signals using ensemble classifiers , 2016, EMBC.

[45]  Adam W. Hoover,et al.  Assessing the Accuracy of a Wrist Motion Tracking Method for Counting Bites Across Demographic and Food Variables , 2017, IEEE Journal of Biomedical and Health Informatics.

[46]  Gary M. Weiss,et al.  Mobile sensor-based biometrics using common daily activities , 2017, 2017 IEEE 8th Annual Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON).

[47]  Lei Jing,et al.  Towards unobtrusive detection and realistic attribute analysis of daily activity sequences using a finger-worn device , 2015, Applied Intelligence.

[48]  Raul I. Ramos-Garcia,et al.  Improving the Recognition of Eating Gestures Using Intergesture Sequential Dependencies , 2015, IEEE Journal of Biomedical and Health Informatics.

[49]  Kingshuk Chakravarty,et al.  A Framework for Pervasive and Ubiquitous Geriatric Monitoring , 2018, HCI.

[50]  Cristina Videira Lopes,et al.  Monitoring Intake Gestures using Sensor Fusion (Microsoft Kinect and Inertial Sensors) for Smart Hom , 2012 .

[51]  John Lach,et al.  Eating gestures detection by tracking finger motion , 2016, 2016 IEEE Wireless Health (WH).

[52]  Christophe G. Giraud-Carrier,et al.  Hand-to-Mouth Motion Tracking in Free-Living Conditions for Improved Weight Control , 2016, 2016 IEEE International Conference on Healthcare Informatics (ICHI).

[53]  Paul Lukowicz,et al.  Gesture spotting with body-worn inertial sensors to detect user activities , 2008, Pattern Recognit..

[54]  Mira Kim,et al.  An analysis of eating activities for automatic food type recognition , 2012, Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference.

[55]  Edward Sazonov,et al.  A Comparative Study of Food Intake Detection Using Artificial Neural Network and Support Vector Machine , 2013, 2013 12th International Conference on Machine Learning and Applications.

[56]  Gerhard Tröster,et al.  Recognition of dietary activity events using on-body sensors , 2008, Artif. Intell. Medicine.

[57]  Peter A. Dinda,et al.  UStress: Understanding college student subjective stress using wrist-based passive sensing , 2017, 2017 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops).

[58]  Adam W. Hoover,et al.  Recognizing Eating Gestures Using Context Dependent Hidden Markov Models , 2016, 2016 IEEE First International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE).

[59]  Abhinav Parate,et al.  Detecting Eating and Smoking Behaviors Using Smartwatches , 2017, Mobile Health - Sensors, Analytic Methods, and Applications.

[60]  Raymond Chiong,et al.  Using Deep Learning and 360 Video to Detect Eating Behavior for User Assistance Systems , 2018, ECIS.

[61]  Adam W. Hoover,et al.  Detecting Eating Using a Wrist Mounted Device During Normal Daily Activities , 2011 .

[62]  Konstantinos Kyritsis,et al.  Automated analysis of in meal eating behavior using a commercial wristband IMU sensor , 2017, 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[63]  Nabil Alshurafa,et al.  When generalized eating detection machine learning models fail in the field , 2017, UbiComp/ISWC Adjunct.

[64]  Mohammad H. Mahoor,et al.  AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild , 2017, IEEE Transactions on Affective Computing.

[65]  Raffaele Limosani,et al.  Enhancing Activity Recognition of Self-Localized Robot Through Depth Camera and Wearable Sensors , 2018, IEEE Sensors Journal.

[66]  Raymond Chiong,et al.  Deep Learning for Human Affect Recognition: Insights and New Developments , 2019, IEEE Transactions on Affective Computing.

[67]  Oliver Amft,et al.  Saving energy on wrist-mounted inertial sensors by motion-adaptive duty-cycling in free-living , 2018, 2018 IEEE 15th International Conference on Wearable and Implantable Body Sensor Networks (BSN).

[68]  Gerhard Tröster,et al.  Probabilistic parsing of dietary activity events , 2007, BSN.

[69]  Vigneshwaran Subbaraju,et al.  Experiences in Building a Real-World Eating Recogniser , 2017, WPA@MobiSys.

[70]  Matthew S. Gerber,et al.  A behavioral sensing system that promotes positive lifestyle changes and improves metabolic control among adults with type 2 diabetes , 2017, 2017 Systems and Information Engineering Design Symposium (SIEDS).

[71]  Surya Sharma,et al.  Automatic Detection of Periods of Eating Using Wrist Motion Tracking , 2016, 2016 IEEE First International Conference on Connected Health: Applications, Systems and Engineering Technologies (CHASE).

[72]  Gregory D. Abowd,et al.  Exploring Symmetric and Asymmetric Bimanual Eating Detection with Inertial Sensors on the Wrist , 2017, DigitalBioMarker@MobiSys.

[73]  Samantha Kleinberg,et al.  Automated estimation of food type and amount consumed from body-worn audio and motion sensors , 2016, UbiComp.

[74]  Ahmad Lotfi,et al.  A Novel Crossings-Based Segmentation Approach for Gesture Recognition , 2018, UKCI.

[75]  Majid Sarrafzadeh,et al.  A Survey of Diet Monitoring Technology , 2017, IEEE Pervasive Computing.

[76]  Nicos Maglaveras,et al.  Real-time Bite Detection from Smartwatch Orientation Sensor Data , 2016, SETN.

[77]  Yang Gao,et al.  Assisting Food Journaling with Automatic Eating Detection , 2016, CHI Extended Abstracts.

[78]  Tobias Alexander Große-Puppendahl,et al.  Enhancing Accelerometer-Based Activity Recognition with Capacitive Proximity Sensing , 2012, AmI.

[79]  Konstantinos Kyritsis,et al.  Personalised meal eating behaviour analysis via semi-supervised learning , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[80]  F. Delpeuch,et al.  Use of variety/diversity scores for diet quality measurement: relation with nutritional status of women in a rural area in Burkina Faso , 2005, European Journal of Clinical Nutrition.

[81]  Vinayak S. Naik,et al.  Using an Arduino and a smartwatch to measure liquid consumed from any container , 2017, 2017 9th International Conference on Communication Systems and Networks (COMSNETS).

[82]  Vigneshwaran Subbaraju,et al.  The case for smartwatch-based diet monitoring , 2015, 2015 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops).

[83]  Paul Lukowicz,et al.  Adapting magnetic resonant coupling based relative positioning technology for wearable activitiy recogniton , 2008, 2008 12th IEEE International Symposium on Wearable Computers.

[84]  David Cornforth,et al.  Exploring the Design of mHealth Systems for Health Behavior Change using Mobile Biosensors , 2019, Commun. Assoc. Inf. Syst..

[85]  Yujie Dong,et al.  A Device for Detecting and Counting Bites of Food Taken by a Person during Eating , 2009, IEEE International Conference on Bioinformatics and Biomedicine.

[86]  Konstantinos Kyritsis,et al.  End-to-end Learning for Measuring in-meal Eating Behavior from a Smartwatch , 2018, 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[87]  Wenyao Xu,et al.  Wearable Food Intake Monitoring Technologies: A Comprehensive Review , 2017, Comput..