From Synsets to Videos: Enriching ItalWordNet Multimodally

The paper describes the multimodal enrichment of ItalWordNet action verbs’ entries by means of an automatic mapping with an ontology of action types instantiated by video scenes (ImagAct). The two resources present important differences as well as interesting complementary features, such that a mapping of these two resources can lead to a an enrichment of IWN, through the connection between synsets and videos apt to illustrate the meaning described by glosses. Here, we describe an approach inspired by ontology matching methods for the automatic mapping of ImagAct video scened onto ItalWordNet sense. The experiments described in the paper are conducted on Italian, but the same methodology can be extended to other languages for which WordNets have been created, since ImagAct is done also for English, Chinese and Spanish. This source of multimodal information can be exploited to design second language learning tools, as well as for language grounding in video action recognition and potentially for robotics.

[1]  Heiner Stuckenschmidt,et al.  Ontology-Based Integration of Information - A Survey of Existing Approaches , 2001, OIS@IJCAI.

[2]  Nicu Sebe,et al.  Distributional semantics with eyes: using image analysis to improve computational representations of word meaning , 2012, ACM Multimedia.

[3]  Deborah L. McGuinness,et al.  An Environment for Merging and Testing Large Ontologies , 2000, KR.

[4]  Robert Lew,et al.  New ways of indicating meaning in electronic dictionaries: hope or hype? , 2009 .

[5]  P. Jaccard.  THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE.1 , 1912 .

[6]  David Sánchez,et al.  Enabling semantic similarity estimation across multiple ontologies: An evaluation in the biomedical domain , 2012, J. Biomed. Informatics.

[7]  A. Tversky.  Features of Similarity , 1977 .

[8]  Gloria Gagliardi,et al.  IMAGACT: Deriving an Action Ontology from Spoken Corpora , 2012, ACL 2012.

[9]  Max J. Egenhofer,et al.  Determining Semantic Similarity among Entity Classes from Different Ontologies , 2003, IEEE Trans. Knowl. Data Eng..

[10]  Mark A. Musen,et al.  Anchor-PROMPT: Using Non-Local Context for Semantic Matching , 2001, OIS@IJCAI.

[11]  Massimo Moneglia,et al.  Mapping a corpus-induced ontology of action verbs on ItalWordNet , 2012 .

[12]  Montse Cuadros,et al.  KnowNet: Building a Large Net of Knowledge from the Web , 2008, COLING.

[13]  Gabriele Stein,et al.  Illustrations in Dictionaries , 1991 .

[14]  Antonietta Alonge,et al.  ItalWordNet: a Large Semantic Database for Italian , 2000, LREC.

[15]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[16]  Francesca Frontini,et al.  Verb interpretation for basic action types: annotation, ontology induction and creation of prototypical scenes , 2012 .