Table Tennis Tutor: Forehand Strokes Classification Based on Multimodal Data and Neural Networks

Beginner table-tennis players require constant real-time feedback while learning the fundamental techniques. However, due to various constraints such as the mentor’s inability to be around all the time, expensive sensors and equipment for sports training, beginners are unable to get the immediate real-time feedback they need during training. Sensors have been widely used to train beginners and novices for various skills development, including psychomotor skills. Sensors enable the collection of multimodal data which can be utilised with machine learning to classify training mistakes, give feedback, and further improve the learning outcomes. In this paper, we introduce the Table Tennis Tutor (T3), a multi-sensor system consisting of a smartphone device with its built-in sensors for collecting motion data and a Microsoft Kinect for tracking body position. We focused on the forehand stroke mistake detection. We collected a dataset recording an experienced table tennis player performing 260 short forehand strokes (correct) and mimicking 250 long forehand strokes (mistake). We analysed and annotated the multimodal data for training a recurrent neural network that classifies correct and incorrect strokes. To investigate the accuracy level of the aforementioned sensors, three combinations were validated in this study: smartphone sensors only, the Kinect only, and both devices combined. The results of the study show that smartphone sensors alone perform sub-par than the Kinect, but similar with better precision together with the Kinect. To further strengthen T3’s potential for training, an expert interview session was held virtually with a table tennis coach to investigate the coach’s perception of having a real-time feedback system to assist beginners during training sessions. The outcome of the interview shows positive expectations and provided more inputs that can be beneficial for the future implementations of the T3.

[1]  Oliver Amft,et al.  COPDTrainer: a smartphone-based motion rehabilitation training system with real-time acoustic feedback , 2013, UbiComp.

[2]  Hendrik Drachsler,et al.  The Big Five: Addressing Recurrent Multimodal Learning Data Challenges , 2018, CrossMMLA@LAK.

[3]  Marcelo Worsley,et al.  Multimodal Learning Analytics and Education Data Mining: using computational technologies to measure complex learning tasks , 2016, J. Learn. Anal..

[4]  Jie Li,et al.  Table Tennis Stroke Recognition Based on Body Sensor Network , 2019, IDCS.

[5]  Björn Eskofier,et al.  Sensor-based stroke detection and stroke type classification in table tennis , 2015, SEMWEB.

[6]  Waraporn Viyanon,et al.  SwingPong: analysis and suggestion based on motion data from mobile sensors for table tennis strokes using decision tree , 2016, ICIIP.

[7]  R W Barnes,et al.  Surgical handicraft: teaching and learning surgical skills. , 1987, American journal of surgery.

[8]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[9]  Hendrik Drachsler,et al.  Multimodal Learning Hub: A Tool for Capturing Customizable Multimodal Learning Experiences , 2018, EC-TEL.

[10]  Hendrik Drachsler,et al.  Dancing Salsa with Machines—Filling the Gap of Dancing Learning Solutions , 2019, Sensors.

[11]  Kalina Yacef,et al.  Let's Dance: How to Build a User Model for Dance Students Using Wearable Technology , 2017, UMAP.

[12]  Akash Anand,et al.  Wearable Audio and IMU Based Shot Detection in Racquet Sports , 2018, ArXiv.

[13]  Herng-Yow Chen,et al.  Real-Time Pitch Training System for Violin Learners , 2012, 2012 IEEE International Conference on Multimedia and Expo Workshops.

[14]  Shenghui Zhao,et al.  A Comparative Study on Human Activity Recognition Using Inertial Sensors in a Smartphone , 2016, IEEE Sensors Journal.

[15]  Hendrik Drachsler,et al.  Read Between the Lines: An Annotation Tool for Multimodal Data for Learning , 2019, LAK.

[16]  Marcus Specht,et al.  Can You Ink While You Blink? Assessing Mental Effort in a Sensor-Based Calligraphy Trainer , 2019, Sensors.

[17]  Paul J. M. Havinga,et al.  Complex Human Activity Recognition Using Smartphone and Wrist-Worn Motion Sensors , 2016, Sensors.

[18]  Antonio Krüger,et al.  Slackliner - An Interactive Slackline Training Assistant , 2018, ACM Multimedia.

[19]  Christian Jutten,et al.  Multimodal Data Fusion: An Overview of Methods, Challenges, and Prospects , 2015, Proceedings of the IEEE.

[20]  Peter van Rosmalen,et al.  Augmenting the Senses: A Review on Sensor-Based Learning Support , 2015, Sensors.

[21]  Thomas G. Dietterich.  Overfitting and undercomputing in machine learning , 1995, CSUR.

[22]  Jooyoung Park,et al.  LSTM-Guided Coaching Assistant for Table Tennis Practice , 2018, Sensors.

[23]  Markus H. Gross,et al.  Combining body sensors and visual sensors for motion training , 2005, ACE '05.

[24]  Md. Kamrul Hasan,et al.  Activity recognition of a badminton game through accelerometer and gyroscope , 2016, 2016 19th International Conference on Computer and Information Technology (ICCIT).

[25]  Hendrik Drachsler,et al.  Detecting Mistakes in CPR Training with Multimodal Data and Neural Networks , 2019, Sensors.

[26]  Peter van Rosmalen,et al.  Can You Help Me with My Pitch? Studying a Tool for Real-Time Automated Feedback , 2016, IEEE Transactions on Learning Technologies.

[27]  Daniela Micucci,et al.  Falls as anomalies? An experimental evaluation using smartphone accelerometer data , 2015, J. Ambient Intell. Humaniz. Comput..

[28]  Saeid Pashazadeh,et al.  Comparative Study of Table Tennis Forehand Strokes Classification Using Deep Learning and SVM , 2020, IEEE Sensors Journal.

[29]  Hendrik Drachsler,et al.  Real-Time Multimodal Feedback with the CPR Tutor , 2020, AIED.