Multimedia analysis and design: a conceptual framework

The paper proposes a conceptual model inspired to semiotic theories, to be applied to the analysis and design of multimedia. We introduce a meta-model with four levels of semantic aggregation. There results a framework of concepts, relations and processes accounting for the multiple meanings that arise from a multimedia text. We explore the effectiveness of the model by considering four commercial clips by the brand Lancôme. Our analysis confirms that the framework is well suited to the analysis, indexing, design of narrative multimedia.

[1]  C. Lauer.  Contending with Terms: “Multimodal” and “Multimedia” in the Academic and Public Spheres , 2009 .

[2]  Kay L. O’Halloran,et al.  Multimodal discourse analysis , 2021, The Bloomsbury Handbook of Discourse Analysis.

[3]  Helen C. Purchase,et al.  Idea Gro up Inc . Copy right Idea Grou p Inc . Copy right Idea Grou p Inc . Copy right Idea Grou p Inc . Chapter I A Semiotic Model of Multimedia : Theory and Evaluation , 2018 .

[4]  P. Verbeek,et al.  Materializing Morality , 2006 .

[5]  Cinzia Bianchi.  Semiotic approaches to advertising texts and strategies: Narrative, passion, marketing , 2011 .

[6]  Daniel Chandler,et al.  Semiotics: The Basics , 2001 .

[7]  Marc Davis,et al.  Knowledge Representation for Video , 1994, AAAI.

[8]  C. Harrison.  Visual Social Semiotics: Understanding How Still Images Make Meaning , 2003 .

[9]  Clarisse Sieckenius deSouza.  The Semiotic Engineering of Human-Computer Interaction (Acting with Technology) , 2005 .

[10]  Theo van Leeuwen,et al.  Reading Images: The Grammar of Visual Design , 1996 .

[11]  Vito Roberto,et al.  Semiotic design and analysis of hypermedia , 2009, HT '09.

[12]  M. Nadin.  Interface design: A semiotic paradigm , 1988 .

[13]  Clarisse Sieckenius de Souza.  Semiotic perspectives on interactive languages for life on the screen , 2013, J. Vis. Lang. Comput..

[14]  Paul Hekkert,et al.  Design for Socially Responsible Behavior: A Classification of Influence Based on Intended User Experience , 2011, Design Issues.

[15]  B. Fogg.  The ethics of persuasive technology , 2003 .

[16]  F. D. Saussure.  Cours de linguistique générale , 1924 .

[17]  J. Bateman,et al.  Towards a grande paradigmatique of film: Christian Metz reloaded , 2007 .

[18]  Davide Bolchini,et al.  Developing heuristics for the semiotics inspection of websites , 2009, SIGDOC '09.

[19]  Johnny Wingstedt,et al.  Narrative Music, Visuals and Meaning in Film , 2010 .

[20]  Vincenzo Lombardo,et al.  Narrative Annotation and Editing of Video , 2010, ICIDS.

[21]  Paloma Díaz,et al.  A Framework for the Analysis and Comparison of Hypermedia Design Methods , 2003, Applied Informatics.

[22]  Jeffrey Bardzell,et al.  Interaction criticism and aesthetics , 2009, CHI.

[23]  Clarisse Sieckenius de Souza.  Semiotic approaches to user interface design , 2001, Knowl. Based Syst..

[24]  John C. McCarthy,et al.  Aesthetics and experience-centered design , 2008, TCHI.

[25]  Lee H. Dowling,et al.  Semiotics and Language: An Analytical Dictionary , 1983 .

[26]  Gilbert Cockton,et al.  Designing worth is worth designing , 2006, NordiCHI '06.

[27]  William C. Mann,et al.  Rhetorical Structure Theory: Toward a functional theory of text organization , 1988 .

[28]  Les Carr,et al.  Rethinking Web Design Models: Requirements for Addressing the Content , 2003 .

[29]  Brian P. Bailey,et al.  SUPPORTING MULTIMEDIA DESIGNERS: TOWARDS MORE EFFECTIVE DESIGN TOOLS , 2001 .

[30]  Y. Behar,et al.  Positive Design: An Introduction to Design for Subjective Well-Being , 2014 .

[31]  Alberto Del Bimbo,et al.  Retrieval of Commercials by Semantic Content: The Semiotic Perspective , 2004, Multimedia Tools and Applications.

[32]  Marcel Worring,et al.  Multimodal Video Indexing : A Review of the State-ofthe-art , 2001 .

[33]  Alan Borning,et al.  Value Sensitive Design and Information Systems , 2020, The Ethics of Information Technologies.

[34]  Harry Bouwman,et al.  Towards user-intuitive web interface sign design and evaluation: A semiotic framework , 2016, Int. J. Hum. Comput. Stud..

[35]  Simon Buckingham Shum,et al.  Cognitive coherence relations and hypertext: from cinematic patterns to scholarly discourse , 2001, HYPERTEXT '01.

[36]  Chiao-i Tseng,et al.  Analysing characters' interactions in filmic text: a functional semiotic approach , 2013 .

[37]  Yvonne Rogers,et al.  New theoretical approaches for human-computer interaction , 2005, Annu. Rev. Inf. Sci. Technol..

[38]  Edward F. McQuarrie,et al.  Visual Rhetoric in Advertising: Text-Interpretive, Experimental, and Reader-Response Analyses , 1999 .

[39]  Henrik Schärfe.  Grand Principles of Narratology , 2007 .

[40]  J. Floch.  Semiotics, Marketing and Communication: Beneath the Signs, the Strategies , 2001 .

[41]  Kaisa Väänänen-Vainio-Mattila,et al.  Value of Information Systems and Products: Understanding the Users’ Perspective and Values , 2009 .

[42]  Louis Hébert,et al.  Louis Hébert , Tools for Text and Image Analysis : An Introduction to Applied Semiotics , 2006 .

[43]  A. Gabrielsson,et al.  The influence of musical structure on emotional expression. , 2001 .

[44]  B. J. Fogg,et al.  Persuasive technology: using computers to change what we think and do , 2002, UBIQ.