A data-driven computational semiotics: The semantic vector space of Magritte’s artworks

Abstract The rise of big digital data is changing the framework within which linguists, sociologists, anthropologists, and other researchers are working. Semiotics is not spared by this paradigm shift. A data-driven computational semiotics is the study with an intensive use of computational methods of patterns in human-created contents related to semiotic phenomena. One of the most promising frameworks in this research program is the Semantic Vector Space (SVS) models and their methods. The objective of this article is to contribute to the exploration of the SVS for a computational semiotics by showing what types of semiotic analysis can be accomplished within this framework. The study is applied to a unique body of digitized artworks. We conducted three short experiments in which we explore three types of semiotic analysis: paradigmatic analysis, componential analysis, and topic modelling analysis. The results reported show that the SVS constitutes a powerful framework within which various types of semiotic analysis can be carried out.

[1]  Luciano Floridi,et al.  Philosophy and Computing: An Introduction , 1999 .

[2]  Dominic Widdows,et al.  Semantic Vector Products: Some Initial Investigations , 2008 .

[3]  Ekaterina Shutova,et al.  Models of Metaphor in NLP , 2010, ACL.

[4]  Alessio Del Bue,et al.  Artistic Image Classification: An Analysis on the PRINTART Database , 2012, ECCV.

[5]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[7]  J. R. Firth,et al.  A Synopsis of Linguistic Theory, 1930-1955 , 1957 .

[8]  Katrin Erk.  Supporting inferences in semantic space: representing words as regions , 2009, IWCS.

[9]  Babak Saleh,et al.  Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature , 2015, ArXiv.

[10]  Steve Kelling,et al.  Data-Intensive Science: A New Paradigm for Biodiversity Studies , 2009 .

[11]  C. J. van Rijsbergen,et al.  The geometry of information retrieval , 2004 .

[12]  Yang Yu,et al.  A Survey of Content-based Image Retrieval , 2002 .

[13]  Alessandro Lenci,et al.  Distributional Memory: A General Framework for Corpus-Based Semantics , 2010, CL.

[14]  ImageNet Classification with Deep Convolutional Neural , 2013 .

[15]  Curt Burgess,et al.  Explorations in context space: Words, sentences, discourse , 1998 .

[16]  David G. Stork,et al.  Computer Vision and Computer Graphics Analysis of Paintings and Drawings: An Introduction to the Literature , 2009, CAIP.

[17]  Curt Burgess,et al.  Producing high-dimensional semantic spaces from lexical co-occurrence , 1996 .

[18]  Ying Liu,et al.  A survey of content-based image retrieval with high-level semantics , 2007, Pattern Recognit..

[19]  Trevor Cohen,et al.  Reasoning with vectors: A continuous model for fast robust inference , 2015, Log. J. IGPL.

[20]  R. Kitchin,et al.  Big Data, new epistemologies and paradigm shifts , 2014, Big Data Soc..

[21]  Kumiko Tanaka-Ishii,et al.  Semiotics of Computing: Filling the Gap Between Humanity and Mechanical Inhumanity , 2015 .

[22]  Michael Felsberg,et al.  Painting-91: a large scale database for computational painting categorization , 2014, Machine Vision and Applications.

[23]  Mihai Nadin,et al.  Information and Semiotic Processes The Semiotics of Computation , 2011, Cybern. Hum. Knowing.

[24]  Chu-Ren Huang,et al.  When Similarity Becomes Opposition: Synonyms and Antonyms Discrimination in DSMs , 2015 .

[25]  A. Greimas,et al.  Figurative Semiotics and the Semiotics of the Plastic Arts , 1989 .

[26]  Lior Shamir,et al.  Computer analysis of art , 2012, JOCCH.

[27]  C. Osgood.  The nature and measurement of meaning. , 1952, Psychological bulletin.

[28]  Victor Pankratius,et al.  Computer-Aided Discovery: Toward Scientific Insight Generation with Machine Support , 2016, IEEE Intelligent Systems.

[29]  Jure Leskovec,et al.  Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change , 2016, ACL.

[30]  Benoît Lemaire,et al.  Effects of High-Order Co-occurrences on Word Semantic Similarities , 2006, ArXiv.

[31]  Ted Dunning,et al.  Accurate Methods for the Statistics of Surprise and Coincidence , 1993, CL.

[32]  Clarisse Sieckenius de Souza,et al.  The Semiotic Engineering of Human-Computer Interaction , 2005 .

[33]  Alexander Mehler,et al.  Methodological Aspects of Computational Semiotics , 2003 .

[34]  L. Shamir.  Computer Analysis Reveals Similarities between the Artistic Styles of Van Gogh and Pollock , 2012, Leonardo.

[35]  David Sylvester,et al.  René Magritte : catalogue raisonné , 1992 .

[36]  Z. Harris,et al.  Methods in structural linguistics. , 1952 .

[37]  Gerlof Bouma,et al.  Normalized (pointwise) mutual information in collocation extraction , 2009 .

[38]  Magnus Sahlgren,et al.  The Distributional Hypothesis , 2008 .

[39]  P. Gärdenfors.  The Geometry of Meaning: Semantics Based on Conceptual Spaces , 2014 .

[40]  T. Landauer,et al.  A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction, and Representation of Knowledge. , 1997 .

[41]  Charles E. Osgood,et al.  Semantic Differential Technique in the Comparative Study of Cultures1 , 2009 .

[42]  D. Kell,et al.  Here is the evidence, now what is the hypothesis? The complementary roles of inductive and hypothesis-driven science in the post-genomic era. , 2004, BioEssays : news and reviews in molecular, cellular and developmental biology.

[43]  Zellig S. Harris,et al.  Distributional Structure , 1954 .

[44]  Damon Mayaffre,et al.  De l'occurrence à l'isotopie. Les co-occurrences en lexicométrie , 2008 .

[45]  Hinrich Schütze,et al.  A Vector Model for Syntagmatic and Paradigmatic Relatedness , 1993 .

[46]  Bryan Pardo,et al.  Classifying paintings by artistic genre: An analysis of features & classifiers , 2009, 2009 IEEE International Workshop on Multimedia Signal Processing.

[47]  Burghard B. Rieger.  Fuzzy Computational Semantics , 1994 .

[48]  Dominic Widdows,et al.  Geometry and Meaning , 2004, Computational Linguistics.

[49]  Patrick Pantel,et al.  From Frequency to Meaning: Vector Space Models of Semantics , 2010, J. Artif. Intell. Res..

[50]  James A. Evans,et al.  Machine Translation: Mining Text for Social Theory , 2016 .

[51]  Natalia Criado,et al.  Intelligent Cybersecurity Agents [Guest editors' introduction] , 2016, IEEE Intell. Syst..

[52]  Ravneet Singh Arora,et al.  Towards automated classification of fine-art painting style: A comparative study , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[53]  J. Meunier.  Vers Une Semiotique Computationnelle , 2018 .

[54]  Mark Steyvers,et al.  Topics in semantic representation. , 2007, Psychological review.

[55]  Peter Gärdenfors,et al.  Conceptual spaces - the geometry of thought , 2000 .

[56]  Magnus Sahlgren,et al.  An Introduction to Random Indexing , 2005 .

[57]  Kai R. Larsen,et al.  9. A Mathematical Approach to Categorization and Labeling of Qualitative Data: The Latent Categorization Method , 2004 .

[58]  Mirella Lapata,et al.  Composition in Distributional Models of Semantics , 2010, Cogn. Sci..

[59]  Dominic Widdows,et al.  Orthogonal Negation in Vector Spaces for Modelling Word-Meanings and Document Retrieval , 2003, ACL.

[60]  P. Rousseeuw.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[61]  James Ze Wang,et al.  Studying digital imagery of ancient paintings by mixtures of stochastic models , 2004, IEEE Transactions on Image Processing.

[62]  Burghard B. Rieger.  Feasible Fuzzy Semantics On Some Problems of How to Handle Word Meaning Empirically , 1981 .

[63]  Lior Wolf,et al.  Classification of Artistic Styles Using Binarized Features Derived from a Deep Neural Network , 2014, ECCV Workshops.

[64]  Jia Li,et al.  Image processing for artist identification , 2008, IEEE Signal Processing Magazine.

[65]  Sung-Hyuk Cha,et al.  The classification of style in fine-art painting , 2005 .

[66]  Jialie Shen,et al.  Stochastic modeling western paintings for effective classification , 2009, Pattern Recognit..

[67]  Jean-Guy Meunier.  Humanités numériques ou computationnelles : Enjeux herméneutiques , 2014 .

[68]  Md. Monirul Islam,et al.  A review on automatic image annotation techniques , 2012, Pattern Recognit..

[69]  Jiro Katto,et al.  Deep Residual Learning for Image Compression , 2019, CVPR Workshops.

[70]  Erez Lieberman Aiden,et al.  Quantitative Analysis of Culture Using Millions of Digitized Books , 2010, Science.

[71]  Magnus Sahlgren,et al.  The Word-Space Model: using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces , 2006 .

[72]  Burghard B. Rieger,et al.  Distributed Semantic Representations of Word Meanings , 1989, Parallelism, Learning, Evolution.

[73]  Lior Shamir,et al.  What makes a Pollock Pollock: a machine vision approach , 2015, Int. J. Arts Technol..

[74]  Kumiko Tanaka-Ishii,et al.  Semiotics of Programming , 2010 .

[75]  Edda Leopold,et al.  On Semantic Spaces , 2005, LDV Forum.

[76]  D. Field,et al.  Mapping the similarity space of paintings: Image statistics and visual perception , 2010 .

[77]  Burghard B. Rieger,et al.  Semiotics and Computational Linguistics , 1999 .

[78]  Curt Burgess,et al.  INVITED REPLY Theory and Operational Definitions in Computational Memory Models: A Response to Glenberg and Robertson , 2000 .

[79]  David M. Mimno,et al.  Computational historiography: Data mining in a century of classics journals , 2012, JOCCH.

[80]  Geoffrey Zweig,et al.  Linguistic Regularities in Continuous Space Word Representations , 2013, NAACL.

[81]  Stephen Clark,et al.  A Systematic Study of Semantic Vector Space Model Parameters , 2014, CVSC@EACL.

[82]  Andrew Zisserman,et al.  The State of the Art: Object Retrieval in Paintings using Discriminative Regions , 2014, BMVC.

[83]  Gerhard Heyer,et al.  A Structuralist Framework for Quantitative Linguistics , 2007, Aspects of Automatic Text Analysis.

[84]  Peter W. Foltz,et al.  An introduction to latent semantic analysis , 1998 .