Digital Twin Coaching for Physical Activities: A Survey

Digital Twin technology has been rising in popularity thanks to the popularity of machine learning in the last decade. As the life expectancy of people around the world is increasing, so is the focus on physical activity to remain healthy especially in the current times where people are staying sedentary while in quarantine. This article aims to provide a survey on the field of Digital Twin technology focusing on machine learning and coaching techniques as they have not been explored yet. We also define what Digital Twin Coaching is and categorize the work done so far in terms of the objective of the physical activity. We also list common Digital Twin Coaching characteristics found in the articles reviewed in terms of concepts such as interactivity, privacy and security and also detail future perspectives in multimodal interaction and standardization, to name a few, that can guide researchers if they choose to work in this field. Finally, we provide a diagram for the Digital Twin Ecosystem showing the interaction between relevant entities and the information flow as well as an idealization of an ideal Digital Twin Ecosystem for team sports’ athlete tracking.

[1]  Suresh Renukappa,et al.  What is a smart device? - a conceptualisation within the paradigm of the internet of things , 2018 .

[2]  Abdulmotaleb El-Saddik,et al.  DIBHR: Depth Image-Based Haptic Rendering , 2008, EuroHaptics.

[3]  Ritu Sadana,et al.  The World report on ageing and health: a policy framework for healthy ageing , 2016, The Lancet.

[4]  Abdulmotaleb El-Saddik,et al.  Development of a wireless CPS for gait parameters measurement and analysis , 2018, 2018 IEEE International Instrumentation and Measurement Technology Conference (I2MTC).

[5]  Brendan Tran Morris,et al.  Learning to Score Olympic Events , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[6]  Goreti Marreiros,et al.  Smartphone Recommendation System to Prevent Potential Injuries in Young Athletes , 2019, IFMBE Proceedings.

[7]  Nouara Achour,et al.  Serious Game based on Skeleton Shape Matching for Functional Rehabilitation Exercises , 2019 .

[8]  Carsten Röcker,et al.  A User-Centered Design Approach to Physical Motion Coaching Systems for Pervasive Health , 2015, Smart Health.

[9]  Tse-Yu Pan,et al.  ORSNet: A Hybrid Neural Network for Official Sports Referee Signal Recognition , 2018, MMSports@MM.

[10]  Masami Nishishiba,et al.  Understanding the relative importance of positive and negative social exchanges: examining specific domains and appraisals. , 2005, The journals of gerontology. Series B, Psychological sciences and social sciences.

[11]  Nicolas Monet,et al.  Lightweight 3D Human Pose Estimation Network Training Using Teacher-Student Learning , 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV).

[12]  Abdulmotaleb El Saddik,et al.  Digital Twins: The Convergence of Multimedia Technologies , 2018, IEEE MultiMedia.

[13]  WangWei,et al.  Recommender system application developments , 2015 .

[14]  A. El Saddik,et al.  EmoJacket: Consumer centric wearable affective jacket to enhance emotional immersion , 2012, 2012 International Conference on Innovations in Information Technology (IIT).

[15]  Jianke Zhu,et al.  AI Coach: Deep Human Pose Estimation and Analysis for Personalized Athletic Training Assistance , 2019, ACM Multimedia.

[16]  Boris Bacic,et al.  Echo State Network Ensemble for Human Motion Data Temporal Phasing: A Case Study on Tennis Forehands , 2016, ICONIP.

[17]  Peijiang Yuan,et al.  Recognition of Yoga Poses Through an Interactive System with Kinect Device , 2018, 2018 2nd International Conference on Robotics and Automation Sciences (ICRAS).

[18]  Keith Mallinson,et al.  Smartphone Revolution: Technology patenting and licensing fosters innovation, market entry, and exceptional growth. , 2015, IEEE Consumer Electronics Magazine.

[19]  Prageet Aeron,et al.  Online Education: Worldwide Status, Challenges, Trends, and Implications , 2018, Journal of Global Information Technology Management.

[20]  Shuxiang Guo,et al.  Study on Motion Recognition for a Hand Rehabilitation Robot Based on sEMG Signals , 2019, 2019 IEEE International Conference on Mechatronics and Automation (ICMA).

[21]  Thurasamy Ramayah,et al.  Wearable technologies: The role of usefulness and visibility in smartwatch adoption , 2016, Comput. Hum. Behav..

[22]  V. C. Padaki,et al.  Smart Vest: wearable multi-parameter remote physiological monitoring system. , 2008, Medical engineering & physics.

[23]  Pasin Israsena,et al.  Comparison of EEG measurement of upper limb movement in motor imagery training system , 2018, BioMedical Engineering OnLine.

[24]  Kiyoung Moon,et al.  The gesture recognition technology based on IMU sensor for personal active spinning , 2018, 2018 20th International Conference on Advanced Communication Technology (ICACT).

[25]  Elena Pozdeeva,et al.  Digital Learning as a Factor of Professional Competitive Growth , 2019 .

[26]  F. Carson,et al.  The Coach–Athlete Relationship in Strength and Conditioning: High Performance Athletes’ Perceptions , 2019, Sports.

[27]  Abdulmotaleb El Saddik,et al.  Standardizing a Shoe Insole Based on ISO/IEEE 11073 Personal Health Device (X73-PHD) Standards , 2019, ICITS.

[28]  Selim Zaim,et al.  A machine learning-based usability evaluation method for eLearning systems , 2013, Decis. Support Syst..

[29]  Marguerite Wotto,et al.  E-learning, M-learning and D-learning: Conceptual definition and comparative analysis , 2018, E-Learning and Digital Media.

[30]  Rongnian Tang,et al.  sEMG Based Gait Phase Recognition for Children with Spastic Cerebral Palsy , 2018, Annals of Biomedical Engineering.

[31]  Shijie Zhang,et al.  Deep Key Frame Extraction for Sport Training , 2017, CCCV.

[32]  Hideki Koike,et al.  FuturePose - Mixed Reality Martial Arts Training Using Real-Time 3D Human Pose Forecasting With a RGB Camera , 2019, 2019 IEEE Winter Conference on Applications of Computer Vision (WACV).

[33]  Tan-Hsu Tan,et al.  Novel IoT-Based Privacy-Preserving Yoga Posture Recognition System Using Low-Resolution Infrared Sensors and Deep Learning , 2019, IEEE Internet of Things Journal.

[34]  Ping Xie,et al.  A virtual rehabilitation system based on EEG-EMG feedback control , 2017, 2017 Chinese Automation Congress (CAC).

[35]  Min Hun Lee A Technology for Computer-Assisted Stroke Rehabilitation , 2018, IUI.

[36]  Dana Kulic,et al.  Exercise motion classification from large-scale wearable sensor data using convolutional neural networks , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[37]  Nikos Ntoumanis,et al.  A review of controlling motivational strategies from a self-determination theory perspective: implications for sports coaches , 2009 .

[38]  Nanning Zheng,et al.  A rehabilitation gait training system for half lower limb disorder , 2017, 2017 Chinese Automation Congress (CAC).

[39]  Andreas Menychtas,et al.  Recommender systems for IoT enabled quantified-self applications , 2019, Evolving Systems.

[40]  Hao Zhang,et al.  An Iterative Method for Classifying Stroke Subjects’ Motor Imagery EEG Data in the BCI-FES Rehabilitation Training System , 2014 .

[41]  Joseph Wei,et al.  How Wearables Intersect with the Cloud and the Internet of Things : Considerations for the developers of wearables. , 2014, IEEE Consumer Electronics Magazine.

[42]  Abdulmotaleb El-Saddik,et al.  Evaluating ALPHAN: A Communication Protocol for Haptic Interaction , 2008, 2008 Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[43]  Stefan Kopp,et al.  Classification of motor errors to provide real-time feedback for sports coaching in virtual reality - A case study in squats and Tai Chi pushes , 2018, Comput. Graph..

[44]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Liqing Zhang,et al.  A Frequency Boosting Method for Motor Imagery EEG Classification in BCI-FES Rehabilitation Training System , 2013, ISNN.

[46]  Simona Crea,et al.  A Wireless Flexible Sensorized Insole for Gait Analysis , 2014, Sensors.

[47]  William Seffens,et al.  Machine learning gesture analysis of yoga for exergame development , 2018, IET Cyper-Phys. Syst.: Theory & Appl..

[48]  Gerhard Fischer,et al.  User Modeling in Human–Computer Interaction , 2001, User Modeling and User-Adapted Interaction.

[49]  Boris Bavci'c,et al.  Towards the next generation of exergames: Flexible and personalised assessment-based identification of tennis swings , 2018, 1804.06948.

[50]  Praminda Caleb-Solly,et al.  Collaborative HRI and Machine Learning for Constructing Personalised Physical Exercise Databases , 2019, TAROS.

[51]  Zülal Güngör,et al.  The usability analysis with heuristic evaluation and analytic hierarchy process , 2009 .

[52]  Steffen Leonhardt,et al.  Wireless and Non-contact ECG Measurement System – the “Aachen SmartChair” , 2007 .

[53]  Franz Konstantin Fuss,et al.  Development of a Smart Insole for Medical and Sports Purposes , 2015 .

[54]  Zanhan Ding,et al.  Rehabilitation system for children with cerebral palsy based on body vector analysis and GMFM-66 standard , 2019, Journal of Physics: Conference Series.

[55]  Carsten Röcker,et al.  Mining crucial features for automatic rehabilitation coaching systems , 2014, PervasiveHealth.

[56]  Julia Fox-Rushby,et al.  A qualitative investigation of the role of sport coaches in designing and delivering a complex community sport intervention for increasing physical activity and improving health , 2018, BMC Public Health.

[57]  Takeo Kanade,et al.  Advances in Multimedia Modeling , 2013, Lecture Notes in Computer Science.

[58]  Andrina Granic,et al.  Experience with usability evaluation of e-learning systems , 2008, Universal Access in the Information Society.

[59]  Mark Wetherell,et al.  The role of coach-athlete relationship quality in team sport athletes’ psychophysiological exhaustion: implications for physical and cognitive performance , 2018, Journal of sports sciences.

[60]  Wei Wang,et al.  Recommender system application developments: A survey , 2015, Decis. Support Syst..

[61]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[62]  Tijana Ivancevic,et al.  Non-linear dynamics in muscle fatigue and strength model during maximal self-perceived elbow extensors training. , 2010, Journal of biomechanics.

[63]  Geneviève A. Mageau,et al.  The coach–athlete relationship: a motivational model , 2003, Journal of sports sciences.

[64]  Lichao Zhang,et al.  A Kinect based Golf Swing Score and Grade System using GMM and SVM , 2012, 2012 5th International Congress on Image and Signal Processing.

[65]  Shih-Ching Yeh,et al.  Hand Rehabilitation Using Virtual Reality Electromyography Signals , 2017, 2017 5th International Conference on Enterprise Systems (ES).

[66]  Yoav Freund,et al.  A decision-theoretic generalization of on-line learning and an application to boosting , 1995, EuroCOLT.

[67]  Ayman Atia,et al.  Smart Coaching: Enhancing Weightlifting and Preventing Injuries , 2019 .

[68]  Maxime Devanne,et al.  Multi-level motion analysis for physical exercises assessment in kinaesthetic rehabilitation , 2017, 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids).

[69]  Abdulmotaleb El Saddik,et al.  An ISO/IEEE 11073 Standardized Digital Twin Framework for Health and Well-Being in Smart Cities , 2020, IEEE Access.

[70]  C.W. Thompson,et al.  Smart Devices and Soft Controllers , 2005, IEEE Internet Comput..

[71]  Ji Chen,et al.  A Novel Telerehabilitation System Based on Bilateral Upper Limb Exoskeleton Robot , 2019, 2019 IEEE International Conference on Real-time Computing and Robotics (RCAR).

[72]  Debi Prosad Dogra,et al.  Virtual trainer with real-time feedback using kinect sensor , 2017, 2017 IEEE Region 10 Symposium (TENSYMP).

[73]  F. O. Isinkaye,et al.  Recommendation systems: Principles, methods and evaluation , 2015 .

[74]  El Saddik,et al.  The Potential of Haptics Technologies , 2007, IEEE Instrumentation & Measurement Magazine.

[75]  A. L. Samuel,et al.  Some studies in machine learning using the game of checkers. II: recent progress , 1967 .

[76]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[77]  Francisco Javier González-Castaño,et al.  SAETA: A Smart Coaching Assistant for Professional Volleyball Training , 2015, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[78]  Bin Sheng,et al.  An Investigation of 3D Human Pose Estimation for Learning Tai Chi: A Human Factor Perspective , 2018, Int. J. Hum. Comput. Interact..

[79]  Abdulmotaleb El-Saddik,et al.  Cardio Twin: A Digital Twin of the human heart running on the edge , 2019, 2019 IEEE International Symposium on Medical Measurements and Applications (MeMeA).

[80]  Chris Button,et al.  Learning to be adaptive as a distributed process across the coach–athlete system: situating the coach in the constraints-led approach , 2018, Physical Education and Sport Pedagogy.

[81]  Edward H. Glaessgen,et al.  The Digital Twin Paradigm for Future NASA and U.S. Air Force Vehicles , 2012 .

[82]  Elena Mugellini,et al.  Designing an e-Coach to Tailor Training Plans for Road Cyclists , 2019, IHSED.

[83]  Björn Eskofier,et al.  Classification of kinematic swimming data with emphasis on resource consumption , 2013, 2013 IEEE International Conference on Body Sensor Networks.

[84]  Hideki Koike,et al.  Real-time Table Tennis Forecasting System based on Long Short-term Pose Prediction Network , 2019, SIGGRAPH Asia Posters.