Improving Viewing Experiences of First-Person Shooter Gameplays with Automatically-Generated Motion Effects

In recent times, millions of people enjoy watching video gameplays at an eSports stadium or home. We seek a method that improves gameplay spectator or viewer experiences by presenting multisensory stimuli. Using a motion chair, we provide the motion effects automatically generated from the audiovisual stream to the viewers watching a first-person shooter (FPS) gameplay. The motion effects express the game character’s movement and gunfire action. We describe algorithms for the computation of such motion effects developed using computer vision techniques and deep learning. By a user study, we demonstrate that our method of providing motion effects significantly improves the viewing experiences of FPS gameplay. The contributions of this paper are with the motion synthesis algorithms integrated for FPS games and the empirical evidence for the benefits of experiencing multisensory gameplays.

[1]  Loïc Cerf,et al.  Watch me playing, i am a professional: a first study on video game live streaming , 2012, WWW.

[2]  Weisi Lin,et al.  Mulsemedia: State of the Art, Perspectives, and Challenges , 2014, TOMM.

[3]  Seungmoon Choi,et al.  Vibrotactile score: A score metaphor for designing vibrotactile patterns , 2009, World Haptics 2009 - Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[4]  Augusto Sarti,et al.  Scream and gunshot detection in noisy environments , 2007, 2007 15th European Signal Processing Conference.

[5]  Bipin Indurkhya,et al.  Towards Multimodal Affective Feedback: Interaction between Visual and Haptic Modalities , 2015, CHI.

[6]  Sang-Ho Shin,et al.  Realistic media authoring tool based on MPEG-V international standard , 2016, 2016 Eighth International Conference on Ubiquitous and Future Networks (ICUFN).

[7]  Seungmoon Choi,et al.  Real-time perception-level translation from audio signals to vibrotactile effects , 2013, CHI.

[8]  Ali Israr,et al.  Tactile brush: drawing on skin with a tactile grid display , 2011, CHI.

[9]  Andrzej Czyzewski,et al.  Detection, classification and localization of acoustic events in the presence of background noise for acoustic surveillance of hazardous situations , 2015, Multimedia Tools and Applications.

[10]  Seungmoon Choi,et al.  Consonance of Vibrotactile Chords , 2014, IEEE Transactions on Haptics.

[11]  Katrien Verbert,et al.  Real-Time Dashboards to Support eSports Spectating , 2018, CHI PLAY.

[12]  Seungmoon Choi,et al.  Improved Haptic Music Player with Auditory Saliency Estimation , 2014, EuroHaptics.

[13]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[14]  Gabriel-Miro Muntean,et al.  Is Multimedia Multisensorial? - A Review of Mulsemedia Systems , 2018, ACM Comput. Surv..

[15]  Antonio Cezar de Castro Lima,et al.  Impulsive Sound Detection Directly in Sigma-Delta Domain , 2017 .

[16]  Heikki Huttunen,et al.  Polyphonic sound event detection using multi label deep neural networks , 2015, 2015 International Joint Conference on Neural Networks (IJCNN).

[17]  Gifford Cheung,et al.  Starcraft from the stands: understanding the game spectator , 2011, CHI.

[18]  Reza Haghighi Osgouei,et al.  Vibrotactile Rendering of Gunshot Events for 4D Films , 2016 .

[19]  Satoshi Tadokoro,et al.  Vibrotactile rendering of camera motion for bimanual experience of first-person view videos , 2017, 2017 IEEE World Haptics Conference (WHC).

[20]  Karon E. MacLean,et al.  Tactile Animation by Direct Manipulation of Grid Displays , 2015, UIST.

[21]  Sang-Kyun Kim,et al.  Authoring multisensorial content , 2013, Signal Process. Image Commun..

[22]  Vincenzo Moscato,et al.  One-Class SVM Based Approach for Detecting Anomalous Audio Events , 2014, 2014 International Conference on Intelligent Networking and Collaborative Systems.

[23]  Soonhung Han,et al.  A framework for automatic creation of motion effects from theatrical motion pictures , 2014, Multimedia Systems.

[24]  Donghee Yvette Wohn,et al.  eSports as An Emerging Research Context at CHI: Diverse Perspectives on Definitions , 2017, CHI Extended Abstracts.

[25]  Carlos Velasco,et al.  Integrating Mid-Air Haptics into Movie Experiences , 2017, TVX.

[26]  Sungkil Lee,et al.  Saliency-Driven Real-Time Video-to-Tactile Translation , 2014, IEEE Transactions on Haptics.

[27]  Ali Israr,et al.  FeelCraft: crafting tactile experiences for media using a feel effect library , 2014, UIST.

[28]  Tom Drummond,et al.  Machine Learning for High-Speed Corner Detection , 2006, ECCV.

[29]  Hayes Raffle,et al.  The sound of touch: physical manipulation of digital sound , 2008, CHI.

[30]  Seungmoon Choi,et al.  Evaluation of vibrotactile pattern design using vibrotactile score , 2012, 2012 IEEE Haptics Symposium (HAPTICS).

[31]  Fausto Pellandini,et al.  Automatic sound detection and recognition for noisy environment , 2000, 2000 10th European Signal Processing Conference.

[32]  Seungmoon Choi,et al.  Tactile Information Transmission by 2D Stationary Phantom Sensations , 2018, CHI.

[33]  Seungmoon Choi,et al.  Seamless Phantom Sensation Moving Across a Wide Range of Body , 2019, 2019 IEEE World Haptics Conference (WHC).

[34]  Karon E. MacLean,et al.  Studying design process and example use with Macaron, a web-based vibrotactile effect editor , 2016, 2016 IEEE Haptics Symposium (HAPTICS).

[35]  Haibo Li,et al.  Turn Your Mobile Into the Ball: Rendering Live Football Game Using Vibration , 2008, IEEE Transactions on Multimedia.

[36]  Kentaro Yoshida,et al.  VibVid: VIBration Estimation from VIDeo by using Neural Network , 2017, ICAT-EGVE.

[37]  Atau Tanaka,et al.  Haptic Wave: A Cross-Modal Interface for Visually Impaired Audio Producers , 2016, CHI.

[38]  Heikki Huttunen,et al.  Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection , 2017, IEEE/ACM Transactions on Audio, Speech, and Language Processing.

[39]  Suranga Nanayakkara,et al.  An enhanced musical experience for the deaf: design and evaluation of a music display and a haptic chair , 2009, CHI.

[40]  Yasushi Ikei,et al.  Rendering of Virtual Walking Sensation by a Passive Body Motion , 2014, EuroHaptics.

[41]  Juho Hamari,et al.  What is eSports and why do people watch it? , 2017, Internet Res..

[42]  Florian Mueller,et al.  Experiencing the Body as Play , 2018, CHI.

[43]  Florian Metze,et al.  A comparison of Deep Learning methods for environmental sound detection , 2017, 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[44]  Sergios Theodoridis,et al.  Gunshot detection in audio streams from movies by means of dynamic programming and Bayesian networks , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[45]  Yüksel Arslan.  Impulsive Sound Detection by a Novel Energy Formula and its Usage for Gunshot Recognition , 2017, ArXiv.

[46]  Robert W. Lindeman,et al.  Haptic ChairIO: A system to study the effect of wind and floor vibration feedback on spatial orientation in VEs , 2015, 2015 IEEE Symposium on 3D User Interfaces (3DUI).

[47]  Stuart Reeves,et al.  Ways of Spectating: Unravelling Spectator Participation in Kinect Play , 2017, CHI.

[48]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[49]  Meyer Nahon,et al.  Simulator motion-drive algorithms - A designer's perspective , 1990 .

[50]  Anatole Lécuyer,et al.  H-Studio: an authoring tool for adding haptic and motion effects to audiovisual content , 2013, UIST.

[51]  Daniel Cremers,et al.  Anisotropic Huber-L1 Optical Flow , 2009, BMVC.

[52]  Buket D. Barkana,et al.  NON-SPEECH ENVIRONMENTAL SOUND CLASSIFICATION USING SVMS WITH A NEW SET OF FEATURES , 2012 .

[53]  W. E. Collins,et al.  Vestibular System Part 2: Psychophysics, Applied Aspects and General Interpretations , 2012, Handbook of Sensory Physiology.

[54]  Yevgeniya Sulema,et al.  Mulsemedia vs. Multimedia: State of the art and future trends , 2016, 2016 International Conference on Systems, Signals and Image Processing (IWSSIP).