A Multimodal System for Real-Time Action Instruction in Motor Skill Learning

We present a multimodal coaching system that supports online motor skill learning. In this domain, closed-loop interaction between the movements of the user and the action instructions by the system is an essential requirement. To achieve this, the actions of the user need to be measured and evaluated and the system must be able to give corrective instructions on the ongoing performance. Timely delivery of these instructions, particularly during execution of the motor skill by the user, is thus of the highest importance. Based on the results of an empirical study on motor skill coaching, we analyze the requirements for an interactive coaching system and present an architecture that combines motion analysis, dialogue management, and virtual human animation in a motion tracking and 3D virtual reality hardware setup. In a preliminary study we demonstrate that the current system is capable of delivering the closed-loop interaction that is required in the motor skill learning domain.

[1]  R. Riener,et al.  Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review , 2012, Psychonomic Bulletin & Review.

[2]  Stefan Kopp,et al.  Multi-Level Analysis of Motor Actions as a Basis for Effective Coaching in Virtual Reality , 2016 .

[3]  Anna Woch,et al.  Using the framework of the kinematic theory for the definition of a movement primitive. , 2004, Motor control.

[4]  Stefan Kopp,et al.  The Behavior Markup Language: Recent Developments and Challenges , 2007, IVA.

[5]  Stefan Kopp,et al.  AsapRealizer 2.0: The Next Steps in Fluent Behavior Realization for ECAs , 2014, IVA.

[6]  Gabriel Skantze,et al.  A General, Abstract Model of Incremental Dialogue Processing , 2009, EACL.

[7]  Tovi Grossman,et al.  YouMove: enhancing movement training with an augmented reality mirror , 2013, UIST.

[8]  Dennis Reidsma,et al.  Continuous interaction with a virtual human , 2011, Journal on Multimodal User Interfaces.

[9]  Stefan Kopp,et al.  Realizing a low-latency virtual reality environment for motor learning , 2015, VRST.

[10]  Ingo Lütkebohle,et al.  A robot as fitness companion: Towards an interactive action-based motivation model , 2014, The 23rd IEEE International Symposium on Robot and Human Interactive Communication.

[11]  R. Schmidt.  A schema theory of discrete motor skill learning. , 1975 .

[12]  Luca Chittaro,et al.  Bringing mobile guides and fitness activities together: a solution based on an embodied virtual trainer , 2006, Mobile HCI.

[13]  Mel Slater,et al.  Taking steps: the influence of a walking technique on presence in virtual reality , 1995, TCHI.

[14]  Stefan Kopp,et al.  An architecture for fluid real-time conversational agents: integrating incremental output generation and input processing , 2013, Journal on Multimodal User Interfaces.

[15]  S. Kopp,et al.  Timing and Grounding in Motor Skill Coaching Interaction: Consequences for the Information State , 2015 .

[16]  Robert S. Kennedy,et al.  Simulator Sickness Questionnaire: An enhanced method for quantifying simulator sickness. , 1993 .

[17]  W. IJsselsteijn,et al.  Virtual Cycling : Effects of Immersion and a Virtual Coach on Motivation and Presence in a Home Fitness Application , 2022 .

[18]  S. Kopp,et al.  Dialogue Structure of Coaching Sessions , 2014 .

[19]  Maja J. Mataric,et al.  Using Socially Assistive Human–Robot Interaction to Motivate Physical Exercise for Older Adults , 2012, Proceedings of the IEEE.

[20]  Gabriel Skantze,et al.  Towards Incremental Speech Generation in Dialogue Systems , 2010, SIGDIAL Conference.

[21]  James F. Allen,et al.  Incremental Dialogue System Faster than and Preferred to its Nonincremental Counterpart , 2007 .