Automatic Annotation Suggestions for Audiovisual Archives: Evaluation Aspects

Abstract In the context of large and ever growing archives, generating annotation suggestions automatically from textual resources related to the documents to be archived is an interesting option in theory. It could save a lot of work in the time consuming and expensive task of manual annotation and it could help cataloguers attain a higher inter-annotator agreement. However, some questions arise in practice: what is the quality of the automatically produced annotations? How do they compare with manual annotations and with the requirements for annotation that were defined in the archive? If different from the manual annotations, are the automatic annotations wrong? In the CHOICE project, partially hosted at the Netherlands Institute for Sound and Vision, the Dutch public archive for audiovisual broadcasts, we automatically generate annotation suggestions for cataloguers. In this paper, we define three types of evaluation of these annotation suggestions: (1) a classic and strict evaluation measure expressing the overlap between automatically generated keywords and the manual annotations, (2) a loosened evaluation measure for which semantically very similar annotations are also considered as relevant matches, and (3) an in-use evaluation of the usefulness of manual versus automatic annotations in the context of serendipitous browsing. During serendipitous browsing, the annotations (manual or automatic) are used to retrieve and visualize semantically related documents.

[1]  Carl Lagoze,et al.  Semantic Browsing , 2003, ECDL.

[2]  Michiel Hildebrand.  Interactive Exploration of Heterogeneous Cultural Heritage Collections , 2008, International Semantic Web Conference.

[3]  Jeff Heflin,et al.  Searching the Web with SHOE , 2000 .

[4]  J. Iria.  T-Rex : A Flexible Relation Extraction Framework , 2004 .

[5]  Hazel K. Bell.  Inter-indexer consistency , 2010 .

[6]  Véronique Malaisé,et al.  Disambiguating automatic semantic annotation based on a thesaurus structure , 2007 .

[7]  Cong Wang,et al.  Keyword Extraction Based on PageRank , 2007, PAKDD.

[8]  H. Simon,et al.  Models of Man. , 1957 .

[9]  Mirja Iivonen,et al.  Consistency in the Selection of Search Concepts and Search Terms , 1995, Information Processing & Management.

[10]  P. Kline.  Models of man , 1986, Nature.

[11]  Susan T. Dumais,et al.  Inductive learning algorithms and representations for text categorization , 1998, CIKM '98.

[12]  Atanas Kiryakov,et al.  Semantic Annotation, Indexing, and Retrieval , 2003, SEMWEB.

[13]  Christian Wartena,et al.  Apolda: A Practical Tool for Semantic Annotation , 2007, 18th International Workshop on Database and Expert Systems Applications (DEXA 2007).

[14]  Ian H. Witten,et al.  Thesaurus-based index term extraction for agricultural documents , 2005 .

[15]  Sergey Brin,et al.  The Anatomy of a Large-Scale Hypertextual Web Search Engine , 1998, Comput. Networks.

[16]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[17]  A. Smeulders,et al.  A Multidisciplinary Approach to Unlocking Television Broadcast Archives , 2009 .

[18]  Marja-Riitta Koivunen,et al.  Annotea: an open RDF infrastructure for shared Web annotations , 2001, WWW '01.

[19]  Kurt Leininger,et al.  Interindexer consistency in PsycINFO , 2000, J. Libr. Inf. Sci..

[20]  R. Likert.  “Technique for the Measurement of Attitudes, A” , 2022, The SAGE Encyclopedia of Research Design.

[21]  Arthur Stutt,et al.  MnM: Ontology Driven Semi-automatic and Automatic Support for Semantic Markup , 2002, EKAW.

[22]  Yorick Wilks,et al.  Designing Adaptive Information Extraction for the Semantic Web in Amilcare , 2003 .