A system for image–text relations in new (and old) media

This article presents a generalized system of image–text relations which applies to different genres of multimodal discourse in which images and texts co-occur. It combines two kinds of relations – the relative status of images and text, and how they relate to one another in terms of logico–semantics. Every instance of an image–text combination in the data sample is described by a selection of features from the system. The units of images and text between which the relations obtain are identified and the realizations of the logico–semantic and status relations are specified, both for the human analyst and a machine. Two application scenarios are discussed. The system should be useful for distinguishing between image–text relations for (genuinely) new and old media.

[1]  Charles Bernstein,et al.  The Visible Word: Experimental Typography and Modern Art, 1909-1923 , 1995 .

[2]  Theo van Leeuwen,et al.  Conjunctive structure in documentary film and television , 1991 .

[3]  KAY O’HALLORAN,et al.  Towards a systemic functional analysis of multisemiotic mathematics texts , 1999 .

[4]  J. Bolter,et al.  Remediation: Understanding New Media , 1999 .

[5]  Johanna Drucker,et al.  The Visible Word: Experimental Typography and Modern Art, 1909-1923 , 1995 .

[6]  Stuart James.  New Media in Late 20th Century Art , 2000 .

[7]  Shi-Kuo Chang,et al.  Visual Languages , 1986, Management and Information Systems.

[8]  L. M. O'Toole.  The language of displayed art , 1994 .

[9]  M. Milwright,et al.  Islamic art and architecture , 2000 .

[10]  Michael Halliday,et al.  An Introduction to Functional Grammar , 1985 .

[11]  Radan Martinec.  The Social Semiotics of Text and Image in Japanese and English Software Manuals and Other Procedures , 2003 .

[12]  Mike Graham,et al.  Linking Video and Text via Representations of Narrative , 2003 .

[13]  Rudolf Arnheim.  The Two Sources of Cognition , 1995 .

[14]  E. Garfield.  Talking science , 1983, Nature.

[15]  Marilyn Domas White,et al.  A taxonomy of relationships between images and text , 2003, J. Documentation.

[16]  Radan Martinec.  Cohesion in action , 1998 .

[17]  Khurshid Ahmad,et al.  “ Talking Pictures : Indexing and Representing Video with Collateral Texts ” , 1998 .

[18]  R. Weiss.  A Taxonomy of Relationships , 1998 .

[19]  Michael Halliday,et al.  Cohesion in English , 1976 .

[20]  Richard A. Watson.  The Picture Theory , 1995 .