Semantic Frames and Visual Scenes: Learning Semantic Role Inventories from Image and Video Descriptions

Frame-semantic parsing and semantic role labelling, that aim to automatically assign semantic roles to arguments of verbs in a sentence, have become an active strand of research in NLP. However, to date these methods have relied on a predefined inventory of semantic roles. In this paper, we present a method to automatically learn argument role inventories for verbs from large corpora of text, images and videos. We evaluate the method against manually constructed role inventories in FrameNet and show that the visual model outperforms the language-only model and operates with a high precision.

[1]  Anna Korhonen,et al.  Improving Verb Clustering with Automatically Acquired Selectional Preferences , 2009, EMNLP.

[2]  Noah A. Smith,et al.  Frame-Semantic Parsing , 2014, CL.

[3]  Carina Silberer,et al.  Learning Grounded Meaning Representations with Autoencoders , 2014, ACL.

[4]  Gemma Boleda,et al.  Distributional Semantics in Technicolor , 2012, ACL.

[5]  Xinlei Chen,et al.  NEIL: Extracting Visual Knowledge from Web Data , 2013, 2013 IEEE International Conference on Computer Vision.

[6]  Christopher R. Johnson,et al.  Background to Framenet , 2003 .

[7]  Peter Young,et al.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions , 2014, TACL.

[8]  Nikhil Garg,et al.  Unsupervised Semantic Role Induction with Global Role Ordering , 2012, ACL.

[9]  Jianbo Shi,et al.  A Random Walks View of Spectral Segmentation , 2001, AISTATS.

[10]  Ali Farhadi,et al.  Learning Everything about Anything: Webly-Supervised Visual Concept Learning , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[11]  P. Resnik.  Selection and information: a class-based approach to lexical relationships , 1993 .

[12]  Ruslan Salakhutdinov,et al.  Multimodal Neural Language Models , 2014, ICML.

[13]  C. Fillmore.  FRAME SEMANTICS AND THE NATURE OF LANGUAGE * , 1976 .

[14]  Stephen Clark,et al.  Exploiting Image Generality for Lexical Entailment Detection , 2015, ACL.

[15]  Xavier Carreras,et al.  Semantic Role Labeling: An Introduction to the Special Issue , 2008, Computational Linguistics.

[16]  Ivan Titov,et al.  Unsupervised Induction of Semantic Roles within a Reconstruction-Error Minimization Framework , 2014, NAACL.

[17]  Michael Wilson,et al.  MRC psycholinguistic database: Machine-usable dictionary, version 2.00 , 1988 .

[18]  Anna Korhonen,et al.  Hierarchical Verb Clustering Using Graph Factorization , 2011, EMNLP.

[19]  Ivan Titov,et al.  A Bayesian Approach to Unsupervised Semantic Role Induction , 2012, EACL.

[20]  Jean Maillard,et al.  Black Holes and White Rabbits: Metaphor Identification with Visual Features , 2016, NAACL.

[21]  Gerard de Melo,et al.  Perceptually Grounded Selectional Preferences , 2015, ACL.

[22]  Mirella Lapata,et al.  Similarity-Driven Semantic Role Induction via Graph Partitioning , 2014, CL.

[23]  Stephen Clark,et al.  Improving Multi-Modal Representations Using Image Dispersion: Why Less is Sometimes More , 2014, ACL.

[24]  Sabine Schulte im Walde,et al.  A Multimodal LDA Model integrating Textual, Cognitive and Visual Modalities , 2013, EMNLP.

[25]  Ted Briscoe,et al.  The Second Release of the RASP System , 2006, ACL.

[26]  Stephen Clark,et al.  Visual Bilingual Lexicon Induction with Transferred ConvNet Features , 2015, EMNLP.

[27]  Mirella Lapata,et al.  Unsupervised Induction of Semantic Roles , 2010, HLT-NAACL.

[28]  Mirella Lapata,et al.  Distributed Representations for Unsupervised Semantic Role Labeling , 2015, EMNLP.

[29]  Sandro Skansi,et al.  Neural Language Models , 2018 .

[30]  Christopher D. Manning,et al.  Unsupervised Discovery of a Statistical Verb Lexicon , 2006, EMNLP.

[31]  Nazli Ikizler-Cinbis,et al.  Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures , 2016, J. Artif. Intell. Res..

[32]  Daniel Gildea,et al.  Automatic Labeling of Semantic Roles , 2000, ACL.