Current and future trends in marine image annotation software

Given the need to describe, analyze and index large quantities of marine imagery data for exploration and monitoring activities, a range of specialized image annotation tools have been developed worldwide. Image annotation – the process of transposing objects or events represented in a video or still image to the semantic level, may involve human interactions and computer-assisted solutions. Marine image annotation software (MIAS) have enabled over 500 publications to date. We review the functioning, application trends and developments, by comparing general and advanced features of 23 different tools utilized in underwater image analysis. MIAS requiring human input are basically a graphical user interface, with a video player or image browser that recognizes a specific time code or image code, allowing to log events in a time-stamped (and/or geo-referenced) manner. MIAS differ from similar software by the capability of integrating data associated to video collection, the most simple being the position coordinates of the video recording platform. MIAS have three main characteristics: annotating events in real time, posteriorly to annotation and interact with a database. These range from simple annotation interfaces, to full onboard data management systems, with a variety of toolboxes. Advanced packages allow to input and display data from multiple sensors or multiple annotators via intranet or internet. Posterior human-mediated annotation often include tools for data display and image analysis, e.g. length, area, image segmentation, point count; and in a few cases the possibility of browsing and editing previous dive logs or to analyze the annotations. The interaction with a database allows the automatic integration of annotations from different surveys, repeated annotation and collaborative annotation of shared datasets, browsing and querying of data. Progress in the field of automated annotation is mostly in post processing, for stable platforms or still images. Integration into available MIAS is currently limited to semi-automated processes of pixel recognition through computer-vision modules that compile expert-based knowledge. Important topics aiding the choice of a specific software are outlined, the ideal software is discussed and future trends are presented.

[1]  Alan F. Smeaton,et al.  Large Scale Evaluations of Multimedia Information Retrieval: The TRECVid Experience , 2005, CIVR.

[2]  Jonas Osterloff,et al.  A computer vision approach for monitoring the spatial and temporal shrimp distribution at the LoVe observatory , 2016 .

[3]  Daniel O.B. Jones,et al.  Deep‐Sea Benthic Sampling , 2013 .

[4]  Tim W. Nattkemper,et al.  DELPHI—fast and adaptive computational laser point detection and visual footprint quantification for arbitrary underwater image collections , 2015, Front. Mar. Sci..

[5]  E. Pante,et al.  Getting to the Point: Accuracy of Point Count in Monitoring Ecosystem Change , 2012 .

[6]  Jörg Ontrup,et al.  Use of machine-learning algorithms for the automated detection of cold-water coral habitats: a pilot study , 2009 .

[7]  Sadegh Abbasi,et al.  Shape similarity retrieval under affine transforms , 2002, Pattern Recognit..

[8]  K. Tanaka,et al.  Efficient management and promotion of utilization of the video information acquired by observation , 2012 .

[9]  Brandon Burr,et al.  VACA: a tool for qualitative video analysis , 2006, CHI Extended Abstracts.

[10]  Tim W. Nattkemper,et al.  Rapid image processing and classification in underwater exploration using advanced high performance computing , 2015, OCEANS 2015 - MTS/IEEE Washington.

[11]  Tim W. Nattkemper,et al.  BIIGLE Tools – A Web 2.0 Approach for Visual Bioimage Database Mining , 2009, 2009 13th International Conference Information Visualisation.

[12]  Dob O. B. Jones,et al.  The use of towed camera platforms in deep-water science , 2009 .

[13]  Daniel O.B. Jones,et al.  A review of the uses of work-class ROVs for the benefits of science: Lessons learned from the SERPENT project , 2005 .

[14]  H. G. Vevers.  Photography of the sea floor , 1951, Journal of the Marine Biological Association of the United Kingdom.

[15]  Whitlow W. L. Au,et al.  Extreme diel horizontal migrations by a tropical nearshore resident micronekton community , 2006 .

[16]  M. Tran,et al.  Mapping and predicting benthic habitats in estuaries using towed underwater video , 2013 .

[17]  N. J. C. Strachan,et al.  Recognition of fish species by colour and shape , 1993, Image Vis. Comput..

[18]  Elizabeth Cook,et al.  Changing coasts: marine aliens and artificial structures , 2012 .

[19]  Francesca Antonucci,et al.  Automated Image Analysis for the Detection of Benthic Crustaceans and Bacterial Mat Coverage Using the VENUS Undersea Cabled Network , 2011, Sensors.

[20]  Federico Pallottino,et al.  Colour calibration for quantitative biological analysis: A novel automated multivariate approach , 2009 .

[21]  Jennifer M. Durden,et al.  A new method for ecological surveying of the abyss using autonomous underwater vehicle photography , 2014 .

[22]  Bert W. Hoeksema,et al.  Global Coordination and Standardisation in Marine Biodiversity through the World Register of Marine Species (WoRMS) and Related Databases , 2013, PloS one.

[23]  Robert B. Fisher,et al.  Automatic fish classification for underwater species behavior understanding , 2010, ARTEMIS '10.

[24]  Christopher J. Smith,et al.  Towards a greater understanding of pattern, scale and process in marine benthic systems: a picture is worth a thousand worms , 2003 .

[25]  Tim W. Nattkemper,et al.  RecoMIA—Recommendations for Marine Image Annotation: Lessons Learned and Future Directions , 2016, Front. Mar. Sci..

[26]  Deva Ramanan,et al.  Efficiently Scaling up Crowdsourced Video Annotation , 2012, International Journal of Computer Vision.

[27]  Christof Koch,et al.  A Model of Saliency-Based Visual Attention for Rapid Scene Analysis , 2009 .

[28]  Andrew R. Maffei,et al.  The Jason II virtual control van system, data acquisition system, web-based event logger, and SeaNet , 2002 .

[29]  Danelle E. Cline,et al.  An Automated Visual Event Detection System for Cabled Observatory Video , 2007, OCEANS 2007.

[30]  M. Hoeberechts,et al.  Detection of salient events in large datasets of underwater video , 2012, 2012 Oceans.

[31]  Nicola L. Foster,et al.  Quality assurance in the identification of deep-sea taxa from video and image analysis: response to Henry and Roberts , 2014 .

[32]  D.R. Edgington,et al.  Detecting, Tracking and Classifying Animals in Underwater Video , 2005, OCEANS 2006.

[33]  K. D. Moore,et al.  Underwater Optical Imaging: Status and Prospects , 2001 .

[34]  Charles H. Zeanah,et al.  Video-taped coding of working model of the child interviews: a viable and useful alternative to verbatim transcripts? , 2004 .

[35]  D.M. Kocak,et al.  Use of a video and laser system to quantify transect area for remotely operated vehicle (ROV) rockfish and abalone surveys , 2005, Proceedings of OCEANS 2005 MTS/IEEE.

[36]  Alexander G. Hauptmann.  Lessons for the Future from a Decade of Informedia Video Analysis Research , 2005, CIVR.

[37]  Md. Monirul Islam,et al.  A review on automatic image annotation techniques , 2012, Pattern Recognit..

[38]  J L Edwards,et al.  Interoperability of biodiversity databases: biodiversity information on every desktop. , 2000, Science.

[39]  Katherine L.C. Bell,et al.  New Frontiers in Ocean Exploration: The E/V Nautilus, NOAA Ship Okeanos Explorer, and R/V Falkor 2016 Field Season , 2017 .

[40]  Karrie Karahalios,et al.  VCode and VData: illustrating a new framework for supporting the video annotation workflow , 2008, AVI '08.

[41]  A. Magurran,et al.  Measuring Biological Diversity , 2004 .

[42]  Malcolm B. Jones,et al.  ROV Image Scaling with Laser Spot Patterns , 2000 .

[43]  Helge Ritter,et al.  AQUISAR: Image Retrieval in Underwater Webcam Images , 2004 .

[44]  Kenneth L. Smith,et al.  Demographic indicators of change in a deposit-feeding abyssal holothurian community (Station M, 4000 m) , 2016 .

[45]  K. Tamburri,et al.  IRL: an Interactive Real-Time, Logging system for ROVs , 2000, OCEANS 2000 MTS/IEEE Conference and Exhibition. Conference Proceedings (Cat. No.00CH37158).

[46]  Mark Rasenberg,et al.  Joint Program Initiative Healthy and productive seas and oceans , 2011 .

[47]  Kevin W Eliceiri,et al.  NIH Image to ImageJ: 25 years of image analysis , 2012, Nature Methods.

[48]  Kenneth L. Smith,et al.  An evaluation of deep-sea benthic megafauna length measurements obtained with laser and stereo camera methods , 2015 .

[49]  Mario Fernando Montenegro Campos,et al.  Particle Filter-Based Predictive Tracking for Robust Fish Counting , 2005, XVIII Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI'05).

[50]  Robert B. Fisher,et al.  Detecting, Tracking and Counting Fish in Low Quality Unconstrained Underwater Videos , 2008, VISAPP.

[51]  Michael Kipp,et al.  ANVIL - a generic annotation tool for multimodal dialogue , 2001, INTERSPEECH.

[52]  Tim W. Nattkemper,et al.  Biigle - Web 2.0 enabled labelling and exploring of images from the Arctic deep-sea observatory HAUSGARTEN , 2009, OCEANS 2009-EUROPE.

[53]  W. Waldo Wakefield,et al.  The use of a Canadian (perspective) grid in deep-sea photography , 1987 .

[54]  E Guillemot,et al.  Video acquisition, archiving, annotation and analysis: NEPTUNE Canada's real-time georeferenced library of deep sea video , 2010, OCEANS 2010 MTS/IEEE SEATTLE.

[55]  Tim W. Nattkemper,et al.  Are we Ready for Science 2.0? , 2012, KMIS.

[56]  F. Grassle.  The Ocean Biogeographic Information System (OBIS): An On-line, Worldwide Atlas for Accessing, Modeling and Mapping Marine Biological Data in a Multidimensional Geographic Context , 2000 .

[57]  Behzad Shahraray,et al.  On the applications of multimedia processing to communications , 1998, Proc. IEEE.

[58]  Ricardo Serrão Santos,et al.  Distribution and habitat association of benthic fish on the Condor seamount (NE Atlantic, Azores) from in situ observations , 2013 .

[59]  James Ze Wang,et al.  Content-based image retrieval: approaches and trends of the new age , 2005, MIR '05.

[60]  B. Shneiderman.  Science 2.0 , 2008, Science.

[61]  Cynthia E. Davies,et al.  EUNIS HABITAT CLASSIFICATION REVISED 2004 , 2004 .

[62]  B.M. Schlining,et al.  MBARI's Video Annotation and Reference System , 2006, OCEANS 2006.

[63]  Pedro Madureira,et al.  Area Estimation of Deep-Sea Surfaces from Oblique Still Images , 2015, PloS one.

[64]  M. Waldrop,et al.  Science 2.0. , 2008, Scientific American.

[65]  J. Gutt,et al.  Semi-Automated Image Analysis for the Assessment of Megafaunal Densities at the Arctic Deep-Sea Observatory HAUSGARTEN , 2012, PloS one.