Mouse Behavior Recognition with The Wisdom of Crowd

In this thesis, we designed and implemented a crowdsourcing system to annotate mouse behaviors in videos; this involves the development of a novel clip-based video labeling tools, that is more efficient than traditional labeling tools in crowdsourcing platform, as well as the design of probabilistic inference algorithms that predict the true labels and the workers' expertise from multiple workers' responses. Our algorithms are shown to perform better than majority vote heuristic. We also carried out extensive experiments to determine the effectiveness of our labeling tool, inference algorithms and the overall system. 3 4 Acknowledgments When Tommy approved my thesis and told me "You did a nice job" a few days ago, I suddenly realized that I would soon conclude my graduate life in MIT. I would like to thank all the people in the Center for Biological & Computational Learning(CBCL) lab, because, in the past two year, I truly enjoy every moment here with you. Especially , I would love to thank Tommy for his supervision and trust in the past two years. Most things would not be possible without his support. I appreciate it more than I can say. Besides, I would like to thank Charles Frogner for helping me with the research and thesis, and training my teamwork skills. Without his help, most of the work in this thesis would riot be possible. He is also the one who uncovers the "secret" undergraduate life in Harvard and anecdotes of Mark Zuckerberg. I am fortunate to have Youssef Mroueh and Guillermo Canas as supportive friends in the past two years. Their intelligence and research attitude have greatly impressed me. They are also great lunch and dinner partners. I also thank Gadi Geiger for his help and wise reminders. He is always the most appropriate person to talk to when I feel in trouble. He is also the best chef and coffee master. Chatting with him every day is more enjoyable than ever. Jim Mutch for coaching me in pool game, Chun-kai Wang for introducing me to snow-boarding, and Kathleen Sullivan for always bring us amazing food/desserts. Besides, I want to thank all my current labmates, Meanwhile, I also thank people I have collaborated with outside the lab over the past two years: Arlene Ducao for creating amazing technical demonstration, Jeremy Scott and Rahul Rajagopalan for winning the MIT Mobile App Competition. They make my life in MIT more exciting …

[1]  Paul A. Viola,et al.  Robust Real-time Object Detection , 2001 .

[2]  Thomas Serre,et al.  Automated home-cage behavioural phenotyping of mice. , 2010, Nature communications.

[3]  Michael H. Dickinson,et al.  TrackFly: Virtual reality for a behavioral system analysis in free-flying fruit flies , 2008, Journal of Neuroscience Methods.

[4]  Pietro Perona,et al.  Automated monitoring and analysis of social behavior in Drosophila , 2009, Nature Methods.

[5]  J. Takahashi,et al.  A robust automated system elucidates mouse home cage behavioral structure , 2008, Proceedings of the National Academy of Sciences.

[6]  Javier R. Movellan,et al.  Whose Vote Should Count More: Optimal Integration of Labels from Labelers of Unknown Expertise , 2009, NIPS.

[7]  M. O. Bumaa,et al.  AUTOMATIC RECOGNITION OF BEHAVIORAL PATTERNS OF RATS USING VIDEO IMAGING AND STATISTICAL CLASSIFICATION , 1998 .

[8]  Xiang D. Tang,et al.  Home cage activity and activity-based measures of anxiety in 129P3/J, 129X1/SvJ and C57BL/6J mice , 2005, Physiology & Behavior.

[9]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[10]  L P Noldus,et al.  EthoVision: A versatile video tracking system for automation of behavioral experiments , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[11]  Ali Farhadi,et al.  The benefits and challenges of collecting richer object annotations , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[12]  David Salesin,et al.  Keyframe-based tracking for rotoscoping and animation , 2004, SIGGRAPH 2004.

[13]  A. J Spink,et al.  The EthoVision video tracking system—A tool for behavioral phenotyping of transgenic mice , 2001, Physiology & Behavior.

[14]  A. P. Dawid,et al.  Maximum Likelihood Estimation of Observer Error‐Rates Using the EM Algorithm , 1979 .

[15]  Shuicheng Yan,et al.  Purposive Hidden-Object-Game: Embedding Human Computation in Popular Game , 2012, IEEE Trans. Multim..

[16]  Devavrat Shah,et al.  Iterative Learning for Reliable Crowdsourcing Systems , 2011, NIPS.

[17]  Jian Dong,et al.  Purposive Hidden-Object-Game: Embedding Human Computation in Popular Game , 2012, IEEE Transactions on Multimedia.

[18]  Thomas C. Henderson,et al.  Video-based Animal Behavior Analysis From Multiple Cameras , 2006, 2006 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems.

[19]  Paul Over,et al.  Evaluation campaigns and TRECVid , 2006, MIR '06.

[20]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[21]  Pietro Perona,et al.  Online crowdsourcing: Rating annotators and obtaining cost-effective labels , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops.

[22]  Paul A. Viola,et al.  Robust Real-Time Face Detection , 2001, International Journal of Computer Vision.

[23]  Serge J. Belongie,et al.  Behavior recognition via sparse spatio-temporal features , 2005, 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance.

[24]  Pietro Perona,et al.  High-throughput Ethomics in Large Groups of Drosophila , 2009, Nature Methods.

[25]  C J Twining,et al.  Robust tracking and posture description for laboratory rodents using active shape models , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[26]  David Mihalcik,et al.  The Design and Implementation of ViPER , 2005 .

[27]  G. Dell’Omo,et al.  Early behavioural changes in mice infected with BSE and scrapie: automated home cage monitoring reveals prion strain differences , 2002, The European journal of neuroscience.

[28]  T. Poggio,et al.  Cognitive neuroscience: Neural mechanisms for the recognition of biological movements , 2003, Nature Reviews Neuroscience.

[29]  William T. Freeman,et al.  On the optimality of solutions of the max-product belief-propagation algorithm in arbitrary graphs , 2001, IEEE Trans. Inf. Theory.

[30]  Robert B. Fisher,et al.  The PETS04 Surveillance Ground-Truth Data Sets , 2004 .

[31]  François Fleuret,et al.  FlowBoost — Appearance learning from sparsely annotated video , 2011, CVPR 2011.

[32]  B. Shukitt-Hale,et al.  Automated measurement of age-related changes in the locomotor response to environmental novelty and home-cage activity , 2001, Mechanisms of Ageing and Development.

[33]  Antonio Torralba,et al.  LabelMe: A Database and Web-Based Tool for Image Annotation , 2008, International Journal of Computer Vision.

[34]  Mark W. Schmidt,et al.  Modeling annotator expertise: Learning when everybody knows a bit of something , 2010, AISTATS.

[35]  Gert R. G. Lanckriet,et al.  A Game-Based Approach for Collecting Semantic Annotations of Music , 2007, ISMIR.

[36]  A. Megens,et al.  Behavioural activity of rats measured by a new method based on the piezo-electric principle , 2004, Psychopharmacology.

[37]  James Hays,et al.  Quality Assessment for Crowdsourced Object Annotations , 2011, BMVC.

[38]  David Salesin,et al.  Keyframe-based tracking for rotoscoping and animation , 2004, ACM Trans. Graph..

[39]  Andrew Steele,et al.  Trainable, vision-based automated home cage behavioral phenotyping , 2010, MB '10.

[40]  Manuel Blum,et al.  reCAPTCHA: Human-Based Character Recognition via Web Security Measures , 2008, Science.

[41]  Antonio Krüger,et al.  Behavioral phenotyping of a murine model of Alzheimer’s disease in a seminaturalistic environment using RFID tracking , 2009, Behavior research methods.

[42]  Thomas C. Henderson,et al.  Constructing Comprehensive Behaviors: A Simulation Study , 2005, CAINE.

[43]  Michael I. Jordan,et al.  Graphical Models, Exponential Families, and Variational Inference , 2008, Found. Trends Mach. Learn..

[44]  Adrian Hilton,et al.  A survey of advances in vision-based human motion capture and analysis , 2006, Comput. Vis. Image Underst..

[45]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[46]  Deva Ramanan,et al.  Efficiently Scaling up Crowdsourced Video Annotation , 2012, International Journal of Computer Vision.

[47]  Sham M. Kakade,et al.  Leveraging archival video for building face datasets , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[48]  Yuichi Motai,et al.  Video-Based Tracking and Incremental Learning Applied to Rodent Behavioral Activity Under Near-Infrared Illumination , 2007, IEEE Transactions on Instrumentation and Measurement.

[49]  The Mouse Phenotype Database Integration Consortium,et al.  The European dimension for the mouse genome mutagenesis program , 2004, Nature Genetics.

[50]  Cordelia Schmid,et al.  Learning realistic human actions from movies , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[51]  Manuel Blum,et al.  Peekaboom: a game for locating objects in images , 2006, CHI.

[52]  R. D'Hooge,et al.  Automatic analysis of altered gait in arylsulphatase A-deficient mice in the open field , 2009, Behavior research methods.

[53]  Nir Friedman,et al.  Probabilistic Graphical Models - Principles and Techniques , 2009 .

[54]  P. Tamborini,et al.  Quantitative analysis of rat activity in the home cage by infrared monitoring. Application to the acute toxicity testing of acetanilide and phenylmercuric acetate , 2004, Archives of Toxicology.

[55]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[56]  Gerardo Hermosillo,et al.  Learning From Crowds , 2010, J. Mach. Learn. Res..

[57]  Frank Dellaert,et al.  MCMC-based particle filtering for tracking a variable number of interacting targets , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[58]  Serge J. Belongie,et al.  Tracking multiple mouse contours (without too many samples) , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[59]  Thomas Serre,et al.  A Biologically Inspired System for Action Recognition , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[60]  Andrew W. Fitzgibbon,et al.  Interactive Feature Tracking using K-D Trees and Dynamic Programming , 2006, 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06).

[61]  Rong Jin,et al.  Learning with Multiple Labels , 2002, NIPS.

[62]  Antonio Torralba,et al.  LabelMe video: Building a video database with human annotations , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[63]  Jian Peng,et al.  Variational Inference for Crowdsourcing , 2012, NIPS.

[64]  Rama Chellappa,et al.  Shape-and-Behavior Encoded Tracking of Bee Dances , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[65]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[66]  Charles R. Severance.  The Apache Software Foundation: Brian Behlendorf , 2012, Computer.

[67]  Laura A. Dabbish,et al.  Labeling images with a computer game , 2004, AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors.

[68]  Pietro Perona,et al.  Social behavior recognition in continuous video , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.