Automatic Annotations and Enrichments for Audiovisual Archives

The practical availability of Audiovisual Processing tools to media scholars and heritage institutions remains limited, despite all the technical advancements of recent years. In this article we present the approach chosen in the CLARIAH project to increase this availability, we discuss the challenges encountered, and introduce the technical solutions we are implementing. Through three use cases focused on the enrichment of AV archives, Pose Analysis, and Automatic Speech Recognition, we demonstrate the potential and breadth of using Audiovisual Processing for archives and Digital Humanities research.

[1]  Franciska de Jong,et al.  Talking with Scholars: Developing a Research Environment for Oral History Collections , 2013, TPDL Workshops.

[2]  Roeland Ordelman,et al.  Easy Listening: Spoken Document Retrieval in CHoral , 2009 .

[3]  Matthew Lincoln,et al.  CAMPI: Computer-Aided Metadata Generation for Photo archives Initiative , 2020 .

[4]  Luigi Marini,et al.  MOVIE: Large Scale Automated Analysis of MOVing ImagEs , 2014, XSEDE '14.

[5]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[6]  Jaap Kamps,et al.  Deep Learning as a Tool for Early Cinema Analysis , 2019, SUMAC @ ACM Multimedia.

[7]  Richard Wright,et al.  Accessing the spoken word , 2005, International Journal on Digital Libraries.

[8]  Victor S. Lempitsky,et al.  Efficient Indexing of Billion-Scale Datasets of Deep Descriptors , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  M. Sheelagh T. Carpendale,et al.  The information flaneur: a fresh look at information seeking , 2011, CHI.

[10]  Roeland Ordelman,et al.  Media Suite: Unlocking Audiovisual Archives for Mixed Media Scholarly Research , 2019 .

[11]  Mark Hedges,et al.  Scholarly primitives: Building institutional infrastructure for humanities e-Science , 2013, Future Gener. Comput. Syst..

[12]  Roeland Ordelman.  Distributed Access to Oral History collections: Fitting Access Technology to the Needs of Collection Owners and Researchers , 2011, DH.

[13]  Lev Manovich.  Visualizing Vertov , 2013 .

[14]  Douglas W. Oard,et al.  Access to recorded interviews: A research agenda , 2008, JOCCH.

[15]  Karen Pearlman,et al.  Cutting Rhythms: Shaping the Film Edit , 2009 .

[16]  Marijn Koolen,et al.  The CLARIAH Media Suite: a Hybrid Approach to System Design in the Humanities , 2019, CHIIR.

[17]  Ellen M. Voorhees,et al.  The TREC Spoken Document Retrieval Track: A Success Story , 2000, TREC.

[18]  Jeffrey Heer,et al.  Narrative Visualization: Telling Stories with Data , 2010, IEEE Transactions on Visualization and Computer Graphics.

[19]  Bhuvana Ramabhadran,et al.  Supporting access to large digital oral history archives , 2002, JCDL '02.

[20]  Mark Liberman,et al.  Transcriber: Development and use of a tool for assisting speech corpora production , 2001, Speech Commun..

[21]  Babak Saleh,et al.  Toward automated discovery of artistic influence , 2014, Multimedia Tools and Applications.

[22]  Melvin Wevers,et al.  The visual digital turn: Using neural networks to study historical images , 2019, Digit. Scholarsh. Humanit..

[23]  Taylor Arnold,et al.  Distant viewing: analyzing large visual corpora , 2019, Digit. Scholarsh. Humanit..

[24]  C. V. Jawahar,et al.  Video retrieval by mimicking poses , 2012, ICMR '12.

[25]  Shawn Graham,et al.  Exploring Big Historical Data: The Historian's Macroscope , 2015 .