Action detection using a neural network elucidates the genetics of mouse grooming behavior

Automated detection of complex animal behaviors remains a challenging problem in neuroscience, particularly for behaviors that consist of disparate sequential motions. Grooming, a prototypical stereotyped behavior, is often used as an endophenotype in psychiatric genetics. Using mouse grooming behavior as an example, we develop a general purpose neural network architecture capable of dynamic action detection at human observer-level performance and operate across dozens of mouse strains with high visual diversity. We provide insights into the amount of human annotated training data that are needed to achieve such performance. We survey grooming behavior in the open field in 2500 mice across 62 strains, determine its heritable components, conduct GWAS to outline its genetic architecture, and perform PheWAS to link human psychiatric traits through shared underlying genetics. Our general machine learning solution that automatically classifies complex behaviors in large datasets will facilitate systematic studies of mechanisms underlying these behaviors.