Beyond Multimedia Authoring

The mulsemedia (Multiple Sensorial Media (MulSeMedia)) concept has been explored to provide users with new sensations using other senses beyond sight and hearing. The demand for producing such applications has motivated various studies in the mulsemedia authoring phase. To encourage researchers to explore new solutions for enhancing the mulsemedia authoring, this survey article reviews several mulsemedia authoring tools and proposals for representing sensory effects and their characteristics. The article also outlines a set of desirable features for mulsemedia authoring tools. Additionally, a multimedia background is discussed to support the proposed study in the mulsemedia field. Open challenges and future directions regarding the mulsemedia authoring phase are also discussed.

[1]  Gabriel-Miro Muntean,et al.  Quality of experience assessment of 3D video synchronised with multisensorial media components , 2017, 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB).

[2]  Yong-Soo Joo,et al.  Sensible Media Simulation in an Automobile Application and Human Responses to Sensory Effects , 2013 .

[3]  Arjeh M. Cohen,et al.  Synchronized Multimedia Integration Language (SMIL) 2.0 , 1998 .

[4]  Peter Radford,et al.  Petri Net Theory and the Modeling of Systems , 1982 .

[5]  Estêvão Bissoli Saleme,et al.  QoE of cross-modally mapped Mulsemedia: an assessment using eye gaze and heart rate , 2020, Multimedia Tools and Applications.

[6]  Dick C. A. Bulterman,et al.  CMIFed: a presentation environment for portable hypermedia documents , 1993, MULTIMEDIA '93.

[7]  Anatole Lécuyer,et al.  H-Studio: an authoring tool for adding haptic and motion effects to audiovisual content , 2013, UIST.

[8]  Débora C. Muchaluat-Saade,et al.  An Approach for Authoring Mulsemedia Documents Based on Events , 2020, 2020 International Conference on Computing, Networking and Communications (ICNC).

[9]  Anas Ali Alkasasbeh,et al.  Using olfactory media cues in e-learning – perspectives from an empirical investigation , 2020, Multimedia Tools and Applications.

[10]  Susanne Boll,et al.  Paving the Last Mile for Multi-Channel Multimedia Presentation Generation , 2005, 11th International Multimedia Modelling Conference.

[11]  Britta Meixner,et al.  Hypervideos and Interactive Multimedia Presentations , 2017, ACM Comput. Surv..

[12]  Fivos Panetsos,et al.  Modeling the Dynamic Behavior of Hypermedia Applications , 2001, IEEE Trans. Software Eng..

[13]  Celso A. S. Santos,et al.  Do I Smell Coffee? The Tale of a 360° Mulsemedia Experience , 2020, IEEE MultiMedia.

[14]  Simone Diniz Junqueira Barbosa,et al.  Extending multimedia languages to support multimodal user interactions , 2017, Multimedia Tools and Applications.

[15]  Michael Wohl,et al.  Final Cut Pro X , 2012 .

[16]  Celso A. S. Santos,et al.  360° Mulsemedia: A Way to Improve Subjective QoE in 360° Videos , 2019, ACM Multimedia.

[17]  李仲.  视音频非线性编辑软件 Adobe Premiere , 1997 .

[18]  Ralf Steinmetz,et al.  A Media Synchronization Survey: Reference Model, Specification, and Case Studies , 1996, IEEE J. Sel. Areas Commun..

[19]  Sang-Kyun Kim,et al.  Authoring multisensorial content , 2013, Signal Process. Image Commun..

[20]  Elisa Bertino,et al.  MPGS: An Interactive Tool for the Specification and Generation of Multimedia Presentations , 2000, IEEE Trans. Knowl. Data Eng..

[21]  Ye.S. Sulema,et al.  TIMEWISE DATA PROCESSING WITH PROGRAMMING LANGUAGE ASAMPL , 2020 .

[22]  Luiz Fernando Gomes Soares,et al.  Composer: Authoring Tool for iTV Programs , 2008, EuroITV.

[23]  I. V. Shevera.  Synchronized multimedia integration language , 2004 .

[24]  Romana Karla Schuler,et al.  Digital Synesthesia: A Model for the Aesthetics of Digital Art , 2016 .

[25]  Yong Soo Joo,et al.  Sensorial Information Extraction and Mapping to Generate Temperature Sensory Effects , 2014 .

[26]  Ntuu,et al.  Semantics and pragmatics of programming language ASAMPL , 2020 .

[27]  D. Stotts,et al.  Trellis: a Formally-defined Hypertextual Basis for Integrating Task and Information , 1999 .

[28]  Débora C. Muchaluat-Saade,et al.  A Hybrid Approach for Spatio-Temporal Validation of Declarative Multimedia Documents , 2018, ACM Trans. Multim. Comput. Commun. Appl..

[29]  Kurt Rothermel,et al.  Representing time in multimedia systems , 1994, 1994 Proceedings of IEEE International Conference on Multimedia Computing and Systems.

[30]  Ferran Marqués,et al.  The MPEG-4 Book , 2003, J. Electronic Imaging.

[31]  Luiz Fernando Gomes Soares,et al.  XConnertor and XTemplate: improving the expressiveness and reuse in web authoring languages , 2002, New Rev. Hypermedia Multim..

[32]  Débora C. Muchaluat-Saade,et al.  NCL4WEB: translating NCL applications to HTML5 web pages , 2013, ACM Symposium on Document Engineering.

[33]  Lloyd Rutledge,et al.  GRiNS: A GRaphical INterface for Creating and Playing SMIL Documents , 1998, Comput. Networks.

[34]  Christian Timmerer,et al.  Improving the Quality of multimedia Experience through sensory effects , 2010, 2010 Second International Workshop on Quality of Multimedia Experience (QoMEX).

[35]  Maria Cristina Ferreira de Oliveira,et al.  A statechart-based model for hypermedia applications , 2001, TOIS.

[36]  Débora C. Muchaluat-Saade,et al.  XTemplate 3.0: spatio-temporal semantics and structure reuse for hypermedia compositions , 2011, Multimedia Tools and Applications.

[37]  Débora C. Muchaluat-Saade,et al.  Mulsemedia preparation: a new event type for preparing media object presentation and sensory effect rendering , 2019, MMSys.

[38]  Chun-Chuan Yang,et al.  Extension of Timeline-based Editing for Non-deterministic Temporal Behavior in SMIL2.0 Authoring , 2008, J. Inf. Sci. Eng..

[39]  조하영,et al.  Event-Based control of 4D effects using MPEG RoSE = MPEG RoSE를 이용한 4D 효과의 이벤트 기반 제어 , 2010 .

[40]  Maximino Bessa,et al.  Collaborative immersive authoring tool for real-time creation of multisensory VR experiences , 2019, Multimedia Tools and Applications.

[41]  Pietro Marchisio,et al.  MediaTouch: A Native Authoring Tool for MHEG-5 Applications , 2004, Multimedia Tools and Applications.

[42]  Jin-Cheon Na,et al.  Dynamic documents: authoring, browsing, and analysis using a high-level petri net-based hypermedia system , 2001, DocEng '01.

[43]  Gabriel-Miro Muntean,et al.  Beyond Multimedia Adaptation: Quality of Experience-Aware Multi-Sensorial Media Delivery , 2015, IEEE Transactions on Multimedia.

[44]  Abdelkader Belkhir,et al.  SMIL builder: An incremental authoring tool for SMIL Documents , 2011, TOMCCAP.

[45]  Carlos André Guimarães Ferraz,et al.  MulSeMaker: An MDD Tool for MulSeMedia Web Application Development , 2017, WebMedia.

[46]  Celso A. S. Santos,et al.  An Event Driven Approach for Integrating Multi-sensory Effects to Interactive Environments , 2015, 2015 IEEE International Conference on Systems, Man, and Cybernetics.

[47]  Silvello Betti,et al.  A novel multimedia-multisensorial 4D platform , 2015, 2015 AEIT International Annual Conference (AEIT).

[48]  Débora C. Muchaluat-Saade,et al.  NEXT: graphical editor for authoring NCL documents supporting composite templates , 2013, EuroITV.

[49]  Celso A. S. Santos,et al.  Crowdsourcing authoring of sensory effects on videos , 2019, Multimedia Tools and Applications.

[50]  Dick C. A. Bulterman,et al.  Structured multimedia authoring , 2005, TOMCCAP.

[51]  Dick C. A. Bulterman,et al.  Authoring Support for Durable Interactive Multimedia Presentations , 1995 .

[52]  Rogério Ferreira Rodrigues,et al.  Modeling, authoring and formatting hypermedia documents in the HyperProp system , 2000, Multimedia Systems.

[53]  Hideo Miyahara,et al.  Multimedia Presentation System Harmony with Temporal and Active Media , 1991, USENIX Summer.

[54]  Pierre de Saqui-Sannes,et al.  HTSPN: an experience in formal modeling of multimedia applications coded in MHEG or Java , 2001 .

[55]  Yevgeniya Sulema,et al.  ASAMPL: Programming Language for Mulsemedia Data Processing Based on Algebraic System of Aggregates , 2017, IMCL.

[56]  Ryuichi Ogawa,et al.  Scenario-Based Hypermedia: A Model and a System , 1992, European Conference on Hypertext.

[57]  Débora C. Muchaluat-Saade,et al.  STEVE: a Hypermedia Authoring Tool based on the Simple Interactive Multimedia Model , 2018, DocEng.

[58]  Gabriel-Miro Muntean,et al.  Is Multimedia Multisensorial? - A Review of Mulsemedia Systems , 2018, ACM Comput. Surv..

[59]  George Ghinea,et al.  Information recall task impact in olfaction-enhanced multimedia , 2013, TOMCCAP.

[60]  Sang-Ho Shin,et al.  Realistic media authoring tool based on MPEG-V international standard , 2016, 2016 Eighth International Conference on Ubiquitous and Future Networks (ICUFN).

[61]  Christian Timmerer,et al.  An end-to-end tool chain for Sensory Experience based on MPEG-V , 2013, Signal Process. Image Commun..

[62]  Celso A. S. Santos,et al.  Less is (Just as Good as) More - an Investigation of Odor Intensity and Hedonic Valence in Mulsemedia QoE using Heart Rate and Eye Tracking , 2020, IEEE Transactions on Multimedia.

[63]  James F. Allen.  Maintaining knowledge about temporal intervals , 1983, CACM.

[64]  Kyoungro Yoon,et al.  Streaming Media with Sensory Effect , 2011, 2011 International Conference on Information Science and Applications.

[65]  Celso A. S. Santos,et al.  Mulsemedia DIY: A Survey of Devices and a Tutorial for Building Your Own Mulsemedia Environment , 2019, ACM Comput. Surv..

[66]  Débora C. Muchaluat-Saade,et al.  Modeling sensory effects as first-class entities in multimedia applications , 2018, MMSys.

[67]  Weisi Lin,et al.  Mulsemedia: State of the Art, Perspectives, and Challenges , 2014, TOMM.

[68]  Polle Zellweger,et al.  Specifying temporal behavior in hypermedia documents , 1992, ECHT '92.

[69]  Christian Timmerer,et al.  Investigating the impact of sensory effects on the Quality of Experience and emotional response in web videos , 2012, 2012 Fourth International Workshop on Quality of Multimedia Experience.

[70]  Dick C. A. Bulterman,et al.  SMIL State: an architecture and implementation for adaptive time-based web applications , 2009, Multimedia Tools and Applications.

[71]  C. Spence.  Crossmodal correspondences: A tutorial review , 2011, Attention, perception & psychophysics.

[72]  Nabil Layaïda,et al.  Madeus, and authoring environment for interactive multimedia documents , 1998, MULTIMEDIA '98.

[73]  Brian P. Bailey,et al.  Nsync—a toolkit for building interactive multimedia presentations , 1998, MULTIMEDIA '98.

[74]  Débora C. Muchaluat-Saade,et al.  Toward Content-Driven Intelligent Authoring of Mulsemedia Applications , 2021, IEEE MultiMedia.

[75]  Ombretta Gaggi,et al.  A visual authoring environment for prototyping multimedia presentations , 2002, Fourth International Symposium on Multimedia Software Engineering, 2002. Proceedings..

[76]  Lynda Hardman,et al.  Modelling and authoring hypermedia documents , 1998 .