Video multitracking of fish behaviour: a synthesis and future perspectives

With the development of digital imaging techniques over the last decade, there are now new opportunities to study complex behavioural patterns in fish (e.g. schooling behaviour) and to track a very large number of individuals. These new technologies and methods provide valuable information to fundamental and applied science disciplines such as ethology, animal sociology, animal psychology, veterinary sciences, animal welfare sciences, statistical physics, pharmacology as well as neuro- and ecotoxicology. This paper presents a review of fish video multitracking techniques. It describes the possibilities of tracking individuals and groups at different scales, but also outlines the advantages and limitations of the detection methods. The problem of occlusions, during which errors of individual identifications are very frequent, is underlined. This paper summarizes different approaches to improving the quality of individual identification, notably by the development of three-dimensional tracking, image analysis and probabilistic applications. Finally, implications for fish research and future directions are presented.

[1]  Craig W. Reynolds Flocks, herds, and schools: a distributed behavioral model , 1987, SIGGRAPH.

[2]  G D Ruxton,et al.  Fish shoal composition: mechanisms and constraints , 2000, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[3]  Ramakant Nevatia,et al.  Tracking multiple humans in crowded environment , 2004, CVPR 2004.

[4]  T. Pitcher,et al.  Fish school density and volume , 1979 .

[5]  I. Aoki A simulation study on the schooling mechanism in fish. , 1982 .

[6]  C. Breder Equations Descriptive of Fish Schools and Other Animal Aggregations , 1954 .

[7]  J. Boal,et al.  Social Behaviour of Individual Oval Squids (Cephalopoda, Teuthoidea, Loliginidae, Sepioteuthis lessoniana) within a Captive School , 2010 .

[8]  Esa Ranta,et al.  Size‐assortative Schooling in Free‐ranging Sticklebacks , 2010 .

[9]  Allan V. Kalueff,et al.  Understanding behavioral and physiological phenotypes of stress and anxiety in zebrafish , 2009, Behavioural Brain Research.

[10]  A. Bisazza,et al.  Ontogeny of Numerical Abilities in Fish , 2010, PloS one.

[11]  I. Couzin,et al.  Effective leadership and decision-making in animal groups on the move , 2005, Nature.

[12]  Noam Miller,et al.  Quantification of shoaling behaviour in zebrafish (Danio rerio) , 2007, Behavioural Brain Research.

[13]  Rui F. Oliveira,et al.  A simple method using a single video camera to determine the three-dimensional position of a fish , 1994 .

[14]  Frank Dellaert,et al.  MCMC Data Association and Sparse Factorization Updating for Real Time Multitarget Tracking with Merged and Multiple Measurements , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  W. Foster,et al.  Group transmission of predator avoidance behaviour in a marine insect: The trafalgar effect , 1981, Animal Behaviour.

[16]  Michael R. Taylor,et al.  Pentylenetetrazole induced changes in zebrafish behavior, neural activity and c-fos expression , 2005, Neuroscience.

[17]  C. Hemelrijk,et al.  Density distribution and size sorting in fish schools: an individual-based model , 2005 .

[18]  Pascal Poncin,et al.  A video multitracking system for quantification of individual behavior in a large fish shoal: Advantages and limits , 2009, Behavior research methods.

[19]  Joseph Katz,et al.  Digital holographic microscopy reveals prey-induced changes in swimming behavior of predatory dinoflagellates , 2007, Proceedings of the National Academy of Sciences.

[20]  D. Weihs Hydromechanics of Fish Schooling , 1973, Nature.

[21]  Frank Dellaert,et al.  MCMC-based particle filtering for tracking a variable number of interacting targets , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[22]  Pascal Poncin,et al.  Comparing the EthoVision 2.3 system and a new computerized multitracking prototype system to measure the swimming behavior in fry fish , 2006, Behavior research methods.

[23]  T. Pitcher Functions of Shoaling Behaviour in Teleosts , 1986 .

[24]  A. Rubinstein,et al.  Zebrafish assays for drug toxicity screening , 2006, Expert opinion on drug metabolism & toxicology.

[25]  J. Watson,et al.  Three‐dimensional spatial coordinates of individual plankton determined using underwater hologrammetry , 2000 .

[26]  F Mondada,et al.  Social Integration of Robots into Groups of Cockroaches to Control Self-Organized Choices , 2007, Science.

[27]  P. Poncin,et al.  Quantifying spontaneous swimming activity in fish with a computerized color video tracking system, a laboratory device using last imaging techniques , 2003, Fish Physiology and Biochemistry.

[28]  R. E. Blaser,et al.  Behavioral measures of anxiety in zebrafish (Danio rerio) , 2010, Behavioural Brain Research.

[29]  N F Hughes,et al.  New techniques for 3-D video tracking of fish swimming movements in still or flowing water , 1996 .

[30]  François Gerlotto,et al.  Three dimensional structure and morphology of pelagic fish schools , 2010 .

[31]  J. Godin,et al.  Predator avoidance and school size in a cyprinodontid fish, the banded killifish (Fundulus diaphanus Lesueur) , 2004, Behavioral Ecology and Sociobiology.

[32]  Tetsu Nemoto,et al.  A computer image processing system for quantification of zebrafish behavior , 2004, Journal of Neuroscience Methods.

[33]  P. Poncin,et al.  Impact implantation of a transmitter on Sarpa salpa behaviour: study with a computerized video tracking system , 2005 .

[34]  V. Gómez,et al.  An automatic colour-based computer vision algorithm for tracking the position of piglets , 2009 .

[35]  R. Gerlai,et al.  Alarm substance induced behavioral responses in zebrafish (Danio rerio) , 2008, Behavioural Brain Research.

[36]  J. Hunter,et al.  Procedure for Analysis of Schooling Behavior , 1966 .

[37]  S. Guo,et al.  Linking genes to brain, behavior and neurological diseases: what can we learn from zebrafish? , 2004, Genes, brain, and behavior.

[38]  J. Krause,et al.  Mixed-species shoaling in fish: the sensory mechanisms and costs of shoal choice , 2002, Behavioral Ecology and Sociobiology.

[39]  C. Anderson Self-Organization in Relation to Several Similar Concepts: Are the Boundaries to Self-Organization Indistinct? , 2002, The Biological Bulletin.

[40]  J. Venkateswara Rao,et al.  Locomotor Behavioral Response of Mosquitofish (Gambusia affinis) to Subacute Mercury Stress Monitored by Video Tracking System , 2007, Drug and chemical toxicology.

[41]  Pascal Poncin,et al.  Cumulative effects of road de-icing salt on amphibian behavior. , 2010, Aquatic toxicology.

[42]  J. Krause,et al.  Social organisation, shoal structure and information transfer , 2003 .

[43]  T. Vicsek,et al.  Collective behavior of interacting self-propelled particles , 2000, cond-mat/0611742.

[44]  David Lentink,et al.  Automated visual tracking for studying the ontogeny of zebrafish swimming , 2008, Journal of Experimental Biology.

[45]  T. Pitcher,et al.  Predator-avoidance behaviours of sand-eel schools: why schools seldom split , 1983 .

[46]  Iain D. Couzin,et al.  The effects of parasitism and body length on positioning within wild fish shoals , 2001 .

[47]  T. J. Pitcher,et al.  A Periscopic Method for Determining the Three-Dimensional Positions of Fish in Schools , 1975 .

[48]  Petter Fossum,et al.  Pretty patterns but a simple strategy: predator- prey interactions between juvenile herring and Atlantic puffins observed with multibeam sonar , 2001 .

[49]  Y. Sawada,et al.  Ontogenetic changes in schooling behaviour during larval and early juvenile stages of Pacific bluefin tuna Thunnus orientalis. , 2010, Journal of fish biology.

[50]  Robert Gerlai,et al.  Zebra Fish: An Uncharted Behavior Genetic Model , 2003, Behavior genetics.

[51]  N. Tinbergen On aims and methods of Ethology , 2010 .

[52]  Ehud Rivlin,et al.  Classification of Moving Targets Based on Motion and Appearance , 2003, BMVC.

[53]  Claire Detrain,et al.  Self-amplification as a source of interindividual variability: shelter selection in cockroaches. , 2009, Journal of insect physiology.

[54]  G. Parisi,et al.  Scale-free correlations in starling flocks , 2009, Proceedings of the National Academy of Sciences.

[55]  D. Brenneman,et al.  Reduction of Submissive Behavior Model for antidepressant drug activity testing: study using a video-tracking system , 2005, Behavioural pharmacology.

[56]  Tsutomu Takagi,et al.  Video analysis of fish schooling behavior in finite space using a mathematical model , 2003 .

[57]  Mathieu Denoël,et al.  The use of visual and automatized behavioral markers to assess methodologies: a study case on PIT-tagging in the Alpine newt , 2011, Behavior research methods.

[58]  R. W. Blake,et al.  Functional design and burst-and-coast swimming in fishes , 1983 .

[59]  H. A. Baldwin,et al.  Methods for measuring the three-dimensional structure of fish schools. , 1965, Animal behaviour.

[60]  L P Noldus,et al.  EthoVision: A versatile video tracking system for automation of behavioral experiments , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[61]  Su Guo,et al.  Preference for ethanol in zebrafish following a single exposure , 2011, Behavioural Brain Research.

[62]  A. S. Kane,et al.  Quantitative movement analysis of social behavior in mummichog, Fundulus heteroclitus , 2006, Journal of Ethology.

[63]  T. J. Pitcher,et al.  The three-dimensional structure of schools in the minnow, Phoxinus phoxinus (L.) , 1973 .

[64]  I. Couzin,et al.  Self-Organization and Collective Behavior in Vertebrates , 2003 .

[65]  George V Lauder,et al.  Hydrodynamics of caudal fin locomotion by chub mackerel, Scomber japonicus (Scombridae). , 2002, The Journal of experimental biology.

[66]  Robert Gerlai,et al.  Oscillations in shoal cohesion in zebrafish (Danio rerio) , 2008, Behavioural Brain Research.

[67]  Christophe Becco,et al.  Experimental evidences of a structural and dynamical transition in fish school , 2006 .

[68]  L. J. Purdy A piece of my mind. Predator and prey. , 1990 .

[69]  Guy Theraulaz,et al.  Self-Organization in Biological Systems , 2001, Princeton studies in complexity.

[70]  Jens Krause,et al.  Shoal choice in zebrafish, Danio rerio: the influence of shoal size and activity , 2001, Animal Behaviour.

[71]  Steven V. Viscido,et al.  Self-Organized Fish Schools: An Examination of Emergent Properties , 2002, The Biological Bulletin.

[72]  Michael Sfakiotakis,et al.  Review of fish swimming modes for aquatic locomotion , 1999 .

[73]  D. Dewsbury,et al.  The proximate and the ultimate: past, present, and future , 1999, Behavioural Processes.

[74]  A. Marshall The Unity of Nature: Wholeness and Disintegration in Ecology and Science , 2002 .

[75]  Steven V. Viscido,et al.  Individual behavior and emergent properties of fish schools: a comparison of observation and theory , 2004 .

[76]  Daniel Cerutti,et al.  Zebrafish provide a sensitive model of persisting neurobehavioral effects of developmental chlorpyrifos exposure: comparison with nicotine and pilocarpine effects and relationship to dopamine deficits. , 2010, Neurotoxicology and teratology.

[77]  A. Peter Klimley,et al.  Schooling in Sphyrna lewini, a Species with Low Risk of Predation: a Non‐egalitarian State , 1985 .

[78]  D. Sumpter,et al.  Inferring the rules of interaction of shoaling fish , 2011, Proceedings of the National Academy of Sciences.

[79]  Hidehiko Okabe,et al.  A method for reconstructing three-dimensional positions of swarming mosquitoes , 1994, Journal of Insect Behavior.

[80]  D. Chivers,et al.  Overriding the oddity effect in mixed-species aggregations: group choice by armored and nonarmored prey , 2003 .

[81]  Pascal Poncin,et al.  Video tracking in the extreme: A new possibility for tracking nocturnal underwater transparent animals with fluorescent elastomer tags , 2011, Behavior research methods.

[82]  Richard E Peterson,et al.  Zebrafish as a model vertebrate for investigating chemical toxicity. , 2005, Toxicological sciences : an official journal of the Society of Toxicology.

[83]  Jens Krause,et al.  Repeated measures of shoaling tendency in zebrafish (Danio rerio) and other small teleost fishes , 2006, Nature Protocols.

[84]  Michael Isard,et al.  Tracking loose-limbed people , 2004, CVPR 2004.

[85]  J. Krause,et al.  Body length assortative shoaling in the European minnow, Phoxinus phoxinus , 2001, Animal Behaviour.

[86]  Pascal Poncin,et al.  Quantifying spontaneous swimming activity in fish with the Ethovision Color-Pro computerized video tracking system, a laboratory device based on digital imaging techniques. , 2003 .

[87]  Jacques Verly,et al.  The State of the Art in Multiple Object Tracking Under Occlusion in Video Sequences , 2003 .

[88]  Joseph Katz,et al.  On the spatial distribution and nearest neighbor distance between particles in the water column determined from in situ holographic measurements , 2006 .

[89]  C. Haddad,et al.  Schooling and swimming behaviors of Hyla semilineata tadpoles (Anura, Hylidae) , 2002 .

[90]  G. Parisi,et al.  Interaction ruling animal collective behavior depends on topological rather than metric distance: Evidence from a field study , 2007, Proceedings of the National Academy of Sciences.

[91]  Siddharth Gaikwad,et al.  Measuring behavioral and endocrine responses to novelty stress in adult zebrafish , 2010, Nature Protocols.

[92]  Charlotte K. Hemelrijk,et al.  Emergence of Oblong School Shape: Models and Empirical Data of Fish , 2010 .

[93]  G. Lauder,et al.  Function of the heterocercal tail in sharks: quantitative wake dynamics during steady horizontal swimming and vertical maneuvering. , 2002, The Journal of experimental biology.

[94]  Adhir K Basu,et al.  Introduction to Stochastic Process , 2002 .

[95]  Charlotte K. Hemelrijk,et al.  Leadership in fish shoals , 2000 .

[96]  Tadashi Inagaki,et al.  Measurements of the three-dimensional structure of free-swimming pelagic fish schools in a natural environment , 1986 .

[97]  School formation and concurrent developmental changes in carangid fish with reference to dietary conditions , 1998 .

[98]  Joseph A. Brown,et al.  A new technique to gather 3-D spatial information using a single camera , 2005 .

[99]  A grid‐net technique for the analysis of fish positions within free‐ranging shoals , 2001 .

[100]  Akira Muto,et al.  Behavioral screening assays in zebrafish. , 2004, Methods in cell biology.

[101]  Magnus Egerstedt,et al.  What Are the Ants Doing? Vision-Based Tracking and Reconstruction of Control Programs , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[102]  A. Manica,et al.  Shoaling preferences in decapod crustacea , 2007, Animal Behaviour.

[103]  S. G. Reebs,et al.  INFLUENCE OF BODY SIZE ON LEADERSHIP IN SHOALS OF GOLDEN SHINERS, NOTEMIGONUS CRYSOLEUCAS , 2001 .

[104]  M. Keenleyside Cichlid fishes : behaviour, ecology and evolution , 1993 .

[105]  M. Keenleyside,et al.  Some Aspects of the Schooling Behaviour of Fish , 1955 .

[106]  Andrew Blake,et al.  A Probabilistic Exclusion Principle for Tracking Multiple Objects , 2000, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[107]  Wei Weng,et al.  Catadioptric stereo-vision system for the real-time monitoring of 3D behavior in aquatic animals , 2007, Physiology & Behavior.

[108]  S. G. Reebs,et al.  Individual leadership and boldness in shoals of golden shiners (Notemigonus crysoleucas) , 2006 .

[109]  W. Stanford,et al.  EOS lentiviral vector selection system for human induced pluripotent stem cells , 2009, Nature Protocols.

[110]  Andrew S. Kane,et al.  Fish models in behavioral toxicology: Automated techniques, updates and perspectives , 2005 .

[111]  I. Couzin,et al.  Inferring the structure and dynamics of interactions in schooling fish , 2011, Proceedings of the National Academy of Sciences.

[112]  Hironobu Fujiyoshi,et al.  Moving target classification and tracking from real-time video , 1998, Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV'98 (Cat. No.98EX201).

[113]  Gregory D. Hager,et al.  Probabilistic Data Association Methods for Tracking Complex Visual Objects , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[114]  S. Nakayama,et al.  Development of schooling behavior in Spanish mackerel Scomberomorus niphonius during early ontogeny , 2003 .

[115]  Rahul Singh,et al.  Automated image-based phenotypic screening for high-throughput drug discovery , 2009, 2009 22nd IEEE International Symposium on Computer-Based Medical Systems.

[116]  Nathaniel K. Newlands,et al.  Measurement of the size, shape and structure of Atlantic bluefin tuna schools in the open ocean , 2008 .

[117]  Kevin N. Laland,et al.  Social learning in fishes : a review , 2003 .

[118]  I. Aoki Experimental and theoretical studies on schooling in fish. II. Internal dynamics of fish schools in relation to inter-fish distance. , 1984 .

[119]  T. Pitcher,et al.  The three-dimensional structure of fish schools , 1980, Behavioral Ecology and Sociobiology.

[120]  C. Lord,et al.  Behavioural phenotyping assays for mouse models of autism , 2010, Nature Reviews Neuroscience.

[121]  J. Krause,et al.  Cross–species familiarity in shoaling fishes , 2003, Proceedings of the Royal Society of London. Series B: Biological Sciences.

[122]  Serge J. Belongie,et al.  Tracking multiple mouse contours (without too many samples) , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[123]  I. Couzin,et al.  Collective memory and spatial sorting in animal groups. , 2002, Journal of theoretical biology.

[124]  Françoise Dibos,et al.  Displacement Following of Hidden Objects in a Video Sequence , 2004, International Journal of Computer Vision.

[125]  Michael Isard,et al.  Contour Tracking by Stochastic Propagation of Conditional Density , 1996, ECCV.

[126]  T. J. Roper,et al.  Group decision-making in animals , 2003, Nature.

[127]  Alexander F. Schier,et al.  Hypocretin/Orexin Overexpression Induces An Insomnia-Like Phenotype in Zebrafish , 2006, The Journal of Neuroscience.

[128]  Michael Isard,et al.  BraMBLe: a Bayesian multiple-blob tracker , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.