Real-Time Style Modelling of Human Locomotion via Feature-Wise Transformations and Local Motion Phases

Controlling the manner in which a character moves in a real-time animation system is a challenging task with useful applications. Existing style transfer systems require access to a reference content motion clip, however, in real-time systems the future motion content is unknown and liable to change with user input. In this work we present a style modelling system that uses an animation synthesis network to model motion content based on local motion phases. An additional style modulation network uses feature-wise transformations to modulate style in real-time. To evaluate our method, we create and release a new style modelling dataset, 100style, containing over 4 million frames of stylised locomotion data in 100 different styles that present a number of challenges for existing systems. To model these styles, we extend the local phase calculation with a contact-free formulation. In comparison to other methods for real-time style modelling, we show our system is more robust and efficient in its style representation while improving motion quality.

[1]  Andrea Vedaldi,et al.  Learning multiple visual domains with residual adapters , 2017, NIPS.

[2]  Jovan Popović,et al.  Style translation for human motion , 2005, ACM Trans. Graph..

[3]  Jessica K. Hodgins,et al.  Realtime style transfer for unlabeled heterogeneous human motion , 2015, ACM Trans. Graph..

[4]  Serge J. Belongie,et al.  Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[5]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[6]  Michael F. Cohen,et al.  Verbs and Adverbs: Multidimensional Motion Interpolation , 1998, IEEE Computer Graphics and Applications.

[7]  Sebastian Starke,et al.  Neural state machine for character-scene interactions , 2019, ACM Trans. Graph..

[8]  Toby P. Breckon,et al.  Real-Time Monocular Depth Estimation Using Synthetic Data with Domain Adaptation via Image Style Transfer , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[9]  Jonas Beskow,et al.  MoGlow , 2019, ACM Trans. Graph..

[10]  Dani Lischinski,et al.  Unpaired motion style transfer from video to animation , 2020, ACM Trans. Graph..

[11]  Yi Zhou,et al.  Auto-Conditioned LSTM Network for Extended Complex Human Motion Synthesis , 2017, ArXiv.

[12]  Amos Storkey,et al.  When Training and Test Sets are Different: Characterising Learning Transfer , 2013 .

[13]  Danica Kragic,et al.  Deep Representation Learning for Human Motion Prediction and Classification , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Aaron C. Courville,et al.  FiLM: Visual Reasoning with a General Conditioning Layer , 2017, AAAI.

[15]  Deok-Kyeong Jang,et al.  Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model , 2021, Proc. ACM Comput. Graph. Interact. Tech..

[16]  Michael J. Black,et al.  On Human Motion Prediction Using Recurrent Neural Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Sebastian Starke,et al.  Local motion phases for learning multi-contact character movements , 2020, ACM Trans. Graph..

[18]  Taku Komura,et al.  Mode-adaptive neural networks for quadruped motion control , 2018, ACM Trans. Graph..

[19]  Michiel van de Panne,et al.  Character controllers using motion VAEs , 2020, ACM Trans. Graph..

[20]  Philipp Slusallek,et al.  Stylistic Locomotion Modeling and Synthesis using Variational Generative Models , 2019, MIG.

[21]  Luigi Gresele,et al.  Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style , 2021, NeurIPS.

[22]  C. Karen Liu,et al.  Learning symmetric and low-energy locomotion , 2018, ACM Trans. Graph..

[23]  Razvan Pascanu,et al.  Test Sample Accuracy Scales with Training Sample Density in Neural Networks , 2021 .

[24]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[25]  Lin Gao,et al.  Autoregressive Stylized Motion Synthesis with Generative Flow , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Sergey Levine,et al.  DeepMimic , 2018, ACM Trans. Graph..

[27]  Ariel Shamir,et al.  Adult2child: Motion Style Transfer using CycleGANs , 2020, MIG.

[28]  Liang Zheng,et al.  Source Free Domain Adaptation with Image Translation , 2020, ArXiv.

[29]  Taku Komura,et al.  Fast Neural Style Transfer for Motion Data , 2017, IEEE Computer Graphics and Applications.

[30]  Ralph Gross,et al.  The CMU Motion of Body (MoBo) Database , 2001 .

[31]  Jitendra Malik,et al.  Recurrent Network Models for Human Dynamics , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[32]  Cristian Sminchisescu,et al.  Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[33]  Dani Lischinski,et al.  Skeleton-aware networks for deep motion retargeting , 2020, ACM Trans. Graph..

[34]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[35]  Christopher K. I. Williams,et al.  Customizing Sequence Generation with Multi-Task Dynamical Systems , 2019, ArXiv.

[36]  Andrea Vedaldi,et al.  Efficient Parametrization of Multi-domain Deep Neural Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[37]  Niloy J. Mitra,et al.  Spectral style transfer for human motion between independent actions , 2016, ACM Trans. Graph..

[38]  Yingying Wang,et al.  Efficient Neural Networks for Real-time Motion Style Transfer , 2019, PACMCGIT.

[39]  Geoffrey E. Hinton,et al.  Factored conditional restricted Boltzmann Machines for modeling motion style , 2009, ICML '09.

[40]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[41]  Jonathon Shlens,et al.  A Learned Representation For Artistic Style , 2016, ICLR.

[42]  Jessica K. Hodgins,et al.  Synthesizing physically realistic human motion in low-dimensional, behavior-specific spaces , 2004, ACM Trans. Graph..

[43]  Jehee Lee,et al.  Interactive character animation by learning multi-objective control , 2018, ACM Trans. Graph..

[44]  Glen Berseth,et al.  DeepLoco , 2017, ACM Trans. Graph..

[45]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[46]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[47]  Taku Komura,et al.  Phase-functioned neural networks for character control , 2017, ACM Trans. Graph..

[48]  Taku Komura,et al.  A Deep Learning Framework for Character Motion Synthesis and Editing , 2016, ACM Trans. Graph..

[49]  Tomohiko Mukai,et al.  Geostatistical motion interpolation , 2005, ACM Trans. Graph..

[50]  Bernhard Schölkopf,et al.  Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration , 2021, ArXiv.

[51]  Oussama Kanoun,et al.  Learned motion matching , 2020, ACM Trans. Graph..

[52]  Taku Komura,et al.  Few‐shot Learning of Homogeneous Human Locomotion Styles , 2018, Comput. Graph. Forum.

[53]  David A. Forsyth,et al.  Generalizing motion edits with Gaussian processes , 2009, ACM Trans. Graph..