The MABe22 Benchmarks for Representation Learning of Multi-Agent Behavior

Real-world behavior is often shaped by complex interactions between multiple agents. To scalably study multi-agent behavior, advances in unsupervised and self-supervised learning have enabled a variety of different behavioral representations to be learned from trajectory data. To date, there does not exist a unified set of benchmarks that can enable comparing methods quantitatively and systematically across a broad set of behavior analysis settings. We aim to address this by introducing a large-scale, multi-agent trajectory dataset from real-world behavioral neuroscience experiments that covers a range of behavior analysis tasks. Our dataset consists of trajectory data from common model organisms, with 9.6 million frames of mouse data and 4.4 million frames of fly data, in a variety of experimental settings, such as different strains, lengths of interaction, and optogenetic stimulation. A subset of the frames also consist of expert-annotated behavior labels. Improvements on our dataset corresponds to behavioral representations that work across multiple organisms and is able to capture differences for common behavior analysis tasks. at-most-yearly community contributions of community contributions dataset for of and annotation data. Community contributions data maintenance plan ensure support for future users of the dataset.

[1]  T. Sproule,et al.  Video based phenotyping platform for the laboratory mouse , 2022 .

[2]  Keith S. Sheppard,et al.  Stride-level analysis of mouse open field behavior using deep-learning-based pose estimation , 2022, Cell reports.

[3]  Jessica L. Verpeut,et al.  Deep phenotyping reveals movement phenotypes in mouse neurodevelopmental models , 2021, Molecular Autism.

[4]  Olivier J. H'enaff,et al.  Perceiver IO: A General Architecture for Structured Inputs & Outputs , 2021, ICLR.

[5]  S. Remy,et al.  Identifying behavioral structure from deep variational embeddings of animal motion , 2020, bioRxiv.

[6]  Timothy W. Dunn,et al.  The PAIR-R24M Dataset for Multi-animal 3D Pose Estimation , 2021, bioRxiv.

[7]  David C. Parkes,et al.  The AI Economist: Optimal Economic Policy Design via Two-level Deep Reinforcement Learning , 2021, ArXiv.

[8]  Swarat Chaudhuri,et al.  Unsupervised Learning of Neurosymbolic Encoders , 2021, ArXiv.

[9]  Paul Pu Liang,et al.  MultiBench: Multiscale Benchmarks for Multimodal Representation Learning , 2021, NeurIPS Datasets and Benchmarks.

[10]  Pietro Perona,et al.  The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions , 2021, NeurIPS Datasets and Benchmarks.

[11]  Serge J. Belongie,et al.  Benchmarking Representation Learning for Natural World Image Collections , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Pietro Perona,et al.  Task Programming: Learning Data Efficient Behavior Representations , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Pietro Perona,et al.  The Mouse Action Recognition System (MARS) software pipeline for automated analysis of social behaviors in mice , 2020, bioRxiv.

[14]  Joelle Pineau,et al.  Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program) , 2020, J. Mach. Learn. Res..

[15]  Yaser Sheikh,et al.  OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Timnit Gebru,et al.  Datasheets for datasets , 2018, Commun. ACM.

[17]  Talmo D. Pereira,et al.  Quantifying behavior to understand the brain , 2020, Nature Neuroscience.

[18]  G. Rubin,et al.  Cell types and neuronal circuitry underlying female aggression in Drosophila , 2020, eLife.

[19]  Jonas Kubilius,et al.  Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence , 2020, Neuron.

[20]  Matthew J. Johnson,et al.  Revealing the structure of pharmacobehavioral space through Motion Sequencing , 2020, Nature Neuroscience.

[21]  Gordon J. Berman,et al.  A framework for studying behavioral evolution by reconstructing ancestral repertoires , 2020, bioRxiv.

[22]  Thomas B. Moeslund,et al.  3D-ZeF: A 3D Zebrafish Tracking Benchmark Dataset , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Mark Chen,et al.  Language Models are Few-Shot Learners , 2020, NeurIPS.

[24]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[25]  Dragomir Anguelov,et al.  Scalability in Perception for Autonomous Driving: Waymo Open Dataset , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[26]  Yisong Yue,et al.  Learning Calibratable Policies using Programmatic Style-Consistency , 2019, ICML.

[27]  Omer Levy,et al.  SpanBERT: Improving Pre-training by Representing and Predicting Spans , 2019, TACL.

[28]  Katja Hofmann.  Minecraft as AI Playground and Laboratory , 2019, CHI PLAY.

[29]  André Susano Pinto,et al.  A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark , 2019, 1910.04867.

[30]  Eric A. Yttri,et al.  B-SOiD: An Open Source Unsupervised Algorithm for Discovery of Spontaneous Behaviors , 2019, bioRxiv.

[31]  Andrew Zisserman,et al.  Video Representation Learning by Dense Predictive Coding , 2019, 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW).

[32]  Ruslan Salakhutdinov,et al.  MineRL: A Large-Scale Dataset of Minecraft Demonstrations , 2019, IJCAI.

[33]  Simon Lucey,et al.  Argoverse: 3D Tracking and Forecasting With Rich Maps , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Jonathan W. Pillow,et al.  Unsupervised identification of the internal states that shape natural behavior , 2019, Nature Neuroscience.

[35]  Abhinav Gupta,et al.  Scaling and Benchmarking Self-Supervised Visual Representation Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[36]  Vivek Kumar,et al.  Robust mouse tracking in complex environments using neural networks , 2018, Communications Biology.

[37]  Dong Liu,et al.  Deep High-Resolution Representation Learning for Human Pose Estimation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Shimon Whiteson,et al.  The StarCraft Multi-Agent Challenge , 2019, AAMAS.

[39]  Alexander Kolesnikov,et al.  Revisiting Self-Supervised Visual Representation Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Yisong Yue,et al.  Generating Multi-Agent Trajectories using Programmatic Weak Supervision , 2018, ICLR.

[41]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[42]  Jesse Davis,et al.  Automatic Discovery of Tactics in Spatio-Temporal Soccer Match Data , 2018, KDD.

[43]  Oriol Vinyals,et al.  Representation Learning with Contrastive Predictive Coding , 2018, ArXiv.

[44]  Sergey Levine,et al.  Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings , 2018, ICML.

[45]  Scott W. Linderman,et al.  The Striatum Organizes 3D Behavior via Moment-to-Moment Action Selection , 2018, Cell.

[46]  David J. Anderson,et al.  The Neuropeptide Tac2 Controls a Distributed Brain State Induced by Chronic Social Isolation Stress , 2018, Cell.

[47]  Nikos Komodakis,et al.  Unsupervised Representation Learning by Predicting Image Rotations , 2018, ICLR.

[48]  Leland McInnes,et al.  UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction , 2018, ArXiv.

[49]  M. Orger,et al.  Structure of the Zebrafish Locomotor Repertoire Revealed with Unsupervised Behavioral Clustering , 2018, Current Biology.

[50]  Frank Hutter,et al.  Fixing Weight Decay Regularization in Adam , 2017, ArXiv.

[51]  Michael B. Reiser,et al.  Mapping the Neural Substrates of Behavior , 2017, Cell.

[52]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[53]  Fabio Viola,et al.  The Kinetics Human Action Video Dataset , 2017, ArXiv.

[54]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Zhiao Huang,et al.  Associative Embedding: End-to-End Learning for Joint Detection and Grouping , 2016, NIPS.

[56]  Pietro Perona,et al.  Learning recurrent representations for hierarchical behavior modeling , 2016, ICLR.

[57]  Ugne Klibaite,et al.  An unsupervised method for quantifying the behavior of paired animals , 2016, Physical biology.

[58]  Michael B. Reiser,et al.  Visual projection neurons in the Drosophila lobula link feature detection to distinct behavioral programs , 2016, eLife.

[59]  Kristin Branson,et al.  Computational Analysis of Behavior. , 2016, Annual review of neuroscience.

[60]  Joshua W. Shaevitz,et al.  Predictability and hierarchy in Drosophila behavior , 2016, Proceedings of the National Academy of Sciences.

[61]  Jia Deng,et al.  Stacked Hourglass Networks for Human Pose Estimation , 2016, ECCV.

[62]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[63]  Ryan P. Adams,et al.  Mapping Sub-Second Structure in Mouse Behavior , 2015, Neuron.

[64]  Ioannis A. Kakadiaris,et al.  A Review of Human Activity Recognition Methods , 2015, Front. Robot. AI.

[65]  David J. Anderson,et al.  Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning , 2015, Proceedings of the National Academy of Sciences.

[66]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[67]  G. Rubin,et al.  Mushroom body output neurons encode valence and guide memory-based action selection in Drosophila , 2014, eLife.

[68]  Yisong Yue,et al.  Learning Fine-Grained Spatial Models for Dynamic Sports Play Prediction , 2014, 2014 IEEE International Conference on Data Mining.

[69]  David J. Anderson,et al.  Toward a Science of Computational Ethology , 2014, Neuron.

[70]  Jonathan Schor,et al.  Detecting Social Actions of Fruit Flies , 2014, ECCV.

[71]  Cristian Sminchisescu,et al.  Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[72]  Stefan R. Pulver,et al.  Independent Optical Excitation of Distinct Neural Populations , 2014, Nature Methods.

[73]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[74]  William Bialek,et al.  Mapping the stereotyped behaviour of freely moving fruit flies , 2013, Journal of The Royal Society Interface.

[75]  Kristin Branson,et al.  JAABA: interactive machine learning for automatic annotation of animal behavior , 2013, Nature Methods.

[76]  Pietro Perona,et al.  Social behavior recognition in continuous video , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[77]  C. Nichols,et al.  Human Disease Models in Drosophila melanogaster and the Role of the Fly in Therapeutic Drug Discovery , 2011, Pharmacological Reviews.

[78]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[79]  D. P. Potasek,et al.  Characterizing fruit fly flight behavior using a microforce sensor with a new comb-drive configuration , 2005, Journal of Microelectromechanical Systems.

[80]  E. Kravitz,et al.  Gender-selective patterns of aggressive behavior in Drosophila melanogaster. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[81]  M. Sokolowski,et al.  Drosophila: Genetics meets behaviour , 2001, Nature Reviews Genetics.

[82]  B. T. Bloomquist,et al.  Isolation of a putative phospholipase c gene of drosophila, norpA, and its role in phototransduction , 1988, Cell.