Machine vision methods for analyzing social interactions

ABSTRACT Recent developments in machine vision methods for automatic, quantitative analysis of social behavior have immensely improved both the scale and level of resolution with which we can dissect interactions between members of the same species. In this paper, we review these methods, with a particular focus on how biologists can apply them to their own work. We discuss several components of machine vision-based analyses: methods to record high-quality video for automated analyses, video-based tracking algorithms for estimating the positions of interacting animals, and machine learning methods for recognizing patterns of interactions. These methods are extremely general in their applicability, and we review a subset of successful applications of them to biological questions in several model systems with very different types of social behaviors. Summary: We review recent developments in machine vision for automatic, quantitative analysis of social behavior that have changed the scale and resolution with which we can dissect interactions between animals.

[1]  W. Hamilton.  The genetical evolution of social behaviour. I. , 1964, Journal of theoretical biology.

[2]  Robert M. Seyfarth,et al.  Recognition of other individuals’ social relationships by female baboons , 1999, Animal Behaviour.

[3]  Serge J. Belongie,et al.  Tracking multiple mouse contours (without too many samples) , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[4]  Kelly M Seagraves,et al.  Evidence for an audience effect in mice: male social partners alter the male vocal response to female cues , 2016, Journal of Experimental Biology.

[5]  J. Todd Streelman,et al.  Evolution of bower building in Lake Malawi cichlid fish: phylogeny, morphology, and behavior , 2015, Front. Ecol. Evol..

[6]  Erick Greene,et al.  Allometry of Alarm Calls: Black-Capped Chickadees Encode Information About Predator Size , 2005, Science.

[7]  Mareike Kritzler,et al.  Emergence of Individuality in Genetically Identical Mice , 2013, Science.

[8]  Michael L. Platt,et al.  Seasonal changes in the structure of rhesus macaque social networks , 2012, Behavioral Ecology and Sociobiology.

[9]  E. Kravitz,et al.  Aggression in Drosophila. , 2015, Behavioral neuroscience.

[10]  Jitendra Malik,et al.  Region-Based Convolutional Networks for Accurate Object Detection and Segmentation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  M. Sokolowski,et al.  Social Interactions in “Simple” Model Systems , 2010, Neuron.

[12]  S. Adamo,et al.  Agonistic behaviour in male and female field crickets, Gryllus bimaculatus, and how behavioural context influences its expression , 1995, Animal Behaviour.

[13]  P. Narins,et al.  Cross-modal integration in a dart-poison frog. , 2005, Proceedings of the National Academy of Sciences of the United States of America.

[14]  Thomas R Insel,et al.  The Challenge of Translation in Social Neuroscience: A Review of Oxytocin, Vasopressin, and Affiliative Behavior , 2010, Neuron.

[15]  Pietro Perona,et al.  High-throughput Ethomics in Large Groups of Drosophila , 2009, Nature Methods.

[16]  Andreas Geiger,et al.  FollowMe: Efficient Online Min-Cost Flow Tracking with Bounded Memory and Computation , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[17]  D. Floreano,et al.  Fluorescence Behavioral Imaging (FBI) Tracks Identity in Heterogeneous Groups of Drosophila , 2012, PloS one.

[18]  Julie H. Simpson,et al.  A suppression hierarchy among competing motor programs drives sequential grooming in Drosophila , 2014, eLife.

[19]  Yi Deng,et al.  Dynamic sensory cues shape song structure in Drosophila , 2014, Nature.

[20]  M. Heisenberg,et al.  Octopamine in Male Aggression of Drosophila , 2008, Current Biology.

[21]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[22]  Hania J. Pavlou,et al.  Courtship behavior in Drosophila melanogaster: towards a ‘courtship connectome’ , 2013, Current Opinion in Neurobiology.

[23]  Michael Dickinson,et al.  The relative roles of vision and chemosensation in mate recognition of Drosophila melanogaster , 2014, Journal of Experimental Biology.

[24]  Joshua P. Neunuebel,et al.  Female mice ultrasonically interact with males during courtship displays , 2015, eLife.

[25]  E. Kravitz,et al.  Scoring and analyzing aggression in Drosophila. , 2012, Cold Spring Harbor protocols.

[26]  R. Kerr,et al.  Discovery of Brainwide Neural-Behavioral Maps via Multiscale Unsupervised Structure Learning , 2014, Science.

[27]  S. Tavaré,et al.  Three-dimensional tracking and behaviour monitoring of multiple fruit flies , 2013, Journal of The Royal Society Interface.

[28]  M. Dickinson,et al.  A New Chamber for Studying the Behavior of Drosophila , 2010, PloS one.

[29]  Elad Schneidman,et al.  High-order social interactions in groups of mice , 2013, eLife.

[30]  A. Crespi,et al.  Tracking Individuals Shows Spatial Fidelity Is a Key Regulator of Ant Social Organization , 2013, Science.

[31]  Kristin Branson,et al.  Understanding classifier errors by examining influential neighbors , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  P. Perona,et al.  utomated multi-day tracking of marked mice for the analysis of ocial behaviour , 2013 .

[33]  M. Dickinson,et al.  Object preference by walking fruit flies, Drosophila melanogaster, is mediated by vision and graviperception , 2010, Journal of Experimental Biology.

[34]  A. Pérez-Escudero,et al.  idTracker: tracking individuals in a group by automatic identification of unmarked animals , 2014, Nature Methods.

[35]  Ullrich Köthe,et al.  Tracking Indistinguishable Translucent Objects over Time Using Weakly Supervised Structured Learning , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[36]  A. Cressant,et al.  Computerized video analysis of social interactions in mice , 2012, Nature Methods.

[37]  Rex A. Kerr,et al.  High-Throughput Behavioral Analysis in C. elegans , 2011, Nature Methods.

[38]  A. Krüger,et al.  Association between exploratory activity and social individuality in genetically identical mice living in the same enriched environment , 2015, Neuroscience.

[39]  Claudio J. Tessone,et al.  How Random Is Social Behaviour? Disentangling Social Complexity through the Study of a Wild House Mouse Population , 2012, PLoS Comput. Biol..

[40]  Aravinthan D. T. Samuel,et al.  Controlling airborne cues to study small animal navigation , 2012, Nature Methods.

[41]  Srinivas C. Turaga,et al.  Mapping social behavior-induced brain activation at cellular resolution in the mouse. , 2014, Cell reports.

[42]  Philipp J. Keller,et al.  Whole-brain functional imaging at cellular resolution using light-sheet microscopy , 2013, Nature Methods.

[43]  P. Anderson.  More is different. , 1972, Science.

[44]  Massimo Piccardi,et al.  Background subtraction techniques: a review , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[45]  Fred A. Hamprecht,et al.  Conservation Tracking , 2013, 2013 IEEE International Conference on Computer Vision.

[46]  P. Pye-Smith.  The Descent of Man, and Selection in Relation to Sex , 1871, Nature.

[47]  David J. Anderson,et al.  P1 interneurons promote a persistent internal state that enhances inter-male aggression in Drosophila , 2015, eLife.

[48]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[49]  R. R. Hoy,et al.  Frequency as a releaser in the courtship song of two crickets, Gryllus bimaculatus (de Geer) and Teleogryllus oceanicus: a neuroethological analysis , 2004, Journal of Comparative Physiology A.

[50]  J. Altmann,et al.  Observational study of behavior: sampling methods. , 1974, Behaviour.

[51]  Philip K. Stoddard,et al.  Recognition of individual neighbors by song in the song sparrow, a species with song repertoires , 1991, Behavioral Ecology and Sociobiology.

[52]  Charless C. Fowlkes,et al.  Globally-optimal greedy algorithms for tracking a variable number of objects , 2011, CVPR 2011.

[53]  Thomas Serre,et al.  Automated home-cage behavioural phenotyping of mice. , 2010, Nature communications.

[54]  Pietro Perona,et al.  Automated monitoring and analysis of social behavior in Drosophila , 2009, Nature Methods.

[55]  W. Kirchner,et al.  Honeybees can be recruited by a mechanical model of a dancing bee , 1989, Naturwissenschaften.

[56]  David J. Anderson,et al.  Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning , 2015, Proceedings of the National Academy of Sciences.

[57]  M. Dow,et al.  Aggression and mating success in Drosophila melanogaster , 1975, Nature.

[58]  William Bialek,et al.  Mapping the stereotyped behaviour of freely moving fruit flies , 2013, Journal of The Royal Society Interface.

[59]  Pascal Fua,et al.  Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Multiple Object Tracking Using K-shortest Paths Optimization , 2022 .

[60]  Ofer Tchernichovski,et al.  A phase plane representation of rat exploratory behavior , 1995, Journal of Neuroscience Methods.

[61]  C.E. Shannon,et al.  Communication in the Presence of Noise , 1949, Proceedings of the IRE.

[62]  M. Koganezawa,et al.  Genes and circuits of courtship behaviour in Drosophila males , 2013, Nature Reviews Neuroscience.

[63]  M. Dickinson,et al.  Social structures depend on innate determinants and chemosensory processing in Drosophila , 2012, Proceedings of the National Academy of Sciences.

[64]  L P Noldus,et al.  EthoVision: A versatile video tracking system for automation of behavioral experiments , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[65]  T. Clutton‐Brock.  Sexual Selection in Males and Females , 2007, Science.

[66]  M. Špinka,et al.  Revisiting play elements and self-handicapping in play: a comparative ethogram of five Old World monkey species. , 2009, Journal of comparative psychology.

[67]  Ann M Graybiel,et al.  Repetitive Behaviors in Monkeys Are Linked to Specific Striatal Activation Patterns , 2004, The Journal of Neuroscience.

[68]  Kristin Branson,et al.  JAABA: interactive machine learning for automatic annotation of animal behavior , 2013, Nature Methods.

[69]  O. Feinerman,et al.  Automated long-term tracking and social behavioural phenotyping of animal colonies within a semi-natural environment , 2013, Nature Communications.