Detecting Biological Locomotion in Video: A Computational Approach

Animals locomote for various reasons: to search for food, to find suitable habitat, to pursue prey, to escape from predators, or to seek a mate. The grand scale of biodiversity contributes to the great locomotory design and mode diversity. In this report, the locomotion of general biological species is referred to as biolocomotion. The goal of this report is to develop a computational approach to detect biolocomotion in any unprocessed video. The ways biological entities locomote through an environment are extremely diverse. Various creatures make use of legs, wings, fins, and other means to move through the world. Significantly, the motion exhibited by the body parts to navigate through an environment can be modelled by a combination of an overall positional advance with an overlaid asymmetric oscillatory pattern, a distinctive signature that tends to be absent in non-biological objects in locomotion. In this report, this key trait of positional advance with asymmetric oscillation along with differences in an object’s common motion (extrinsic motion) and localized motion of its parts (intrinsic motion) is exploited to detect biolocomotion. In particular, a computational algorithm is developed to measure the presence of these traits in tracked objects to determine if they correspond to a biological entity in locomotion. An alternative algorithm, based on generic handcrafted features combined with learning is assembled out of components from allied areas of investigation, also is presented as a basis of comparison to the main proposed algorithm. A novel biolocomotion dataset encompassing a wide range of moving biological and non-biological objects in natural settings is provided. Also, biolocomotion annotations to an extant camouflage animals dataset is provided. Quantitative results indicate that the proposed algorithm considerably outperforms the alternative approach, supporting the hypothesis that biolocomotion can be detected reliably based on its distinct signature of positional advance with asymmetric oscillation and extrinsic/intrinsic motion dissimilarity.

[1]  Bernard Ghanem,et al.  ActivityNet: A large-scale video benchmark for human activity understanding , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Lior Wolf,et al.  Live Repetition Counting , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[3]  J. P. Jones,et al.  An evaluation of the two-dimensional Gabor filter model of simple receptive fields in cat striate cortex. , 1987, Journal of neurophysiology.

[4]  George V. Lauder,et al.  Fish Locomotion: Biology and Robotics of Body and Fin-Based Movements , 2015 .

[5]  Jiang Yu Zheng,et al.  Predict Vehicle Collision by TTC From Motion Using a Single Video Camera , 2018, IEEE Transactions on Intelligent Transportation Systems.

[6]  A. F. O'connell,et al.  Camera traps in animal ecology : methods and analyses , 2011 .

[7]  S. Lea,et al.  Categorization of natural movements by pigeons: visual concept discrimination and biological motion. , 1998, Journal of the experimental analysis of behavior.

[8]  Chenliang Xu,et al.  Can humans fly? Action understanding with multiple classes of actors , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Ali Borji,et al.  Salient Object Detection: A Benchmark , 2015, IEEE Transactions on Image Processing.

[10]  Mubarak Shah,et al.  Cyclic motion detection for motion based recognition , 1994, Pattern Recognit..

[11]  Yael Pritch,et al.  Saliency filters: Contrast based filtering for salient region detection , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[12]  Yang Song,et al.  The iNaturalist Species Classification and Detection Dataset , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[13]  J. Cutting,et al.  Recognizing the sex of a walker from a dynamic point-light display , 1977 .

[14]  Edward H. Adelson,et al.  Analyzing and recognizing walking figures in XYT , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Nanning Zheng,et al.  Learning to Detect a Salient Object , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  N. Troje,et al.  The Inversion Effect in Biological Motion Perception: Evidence for a “Life Detector”? , 2006, Current Biology.

[17]  E. Shavit,et al.  Motion understanding from qualitative visual dynamics , 1993, [1993] Proceedings IEEE Workshop on Qualitative Vision.

[18]  Mark H Johnson,et al.  Biological Motion: A Perceptual Life Detector? , 2006, Current Biology.

[19]  Li Wen,et al.  Understanding undulatory locomotion in fishes using an inertia-compensated flapping foil robotic device , 2013, Bioinspiration & biomimetics.

[20]  Martial Hebert,et al.  Event Detection in Crowded Videos , 2007, 2007 IEEE 11th International Conference on Computer Vision.

[21]  Chris P. Tsokos,et al.  Mathematical Statistics with Applications , 2009 .

[22]  Cordelia Schmid,et al.  Towards Understanding Action Recognition , 2013, 2013 IEEE International Conference on Computer Vision.

[23]  Narendra Ahuja,et al.  Extraction and Analysis of Multiple Periodic Motions in Video Sequences , 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Larry S. Davis,et al.  Pedestrian Detection via Periodic Motion Analysis , 2007, International Journal of Computer Vision.

[25]  Shi-Min Hu,et al.  Global contrast based salient region detection , 2011, CVPR 2011.

[26]  Jake K. Aggarwal,et al.  Human Motion Analysis: A Review , 1999, Comput. Vis. Image Underst..

[27]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[28]  T. Poggio,et al.  A network that learns to recognize three-dimensional objects , 1990, Nature.

[29]  Mohieddine Jelali,et al.  Detection of Oscillating Control Loops , 2013 .

[30]  G. Mather,et al.  Recognition of Animal Locomotion from Dynamic Point-Light Displays , 1993, Perception.

[31]  R. Blickhan,et al.  Locomotion Energetics of the Ghost Crab: II. Mechanics of the Centre of Mass During Walking and Running , 1987 .

[32]  Cordelia Schmid,et al.  Action Recognition with Improved Trajectories , 2013, 2013 IEEE International Conference on Computer Vision.

[33]  Dorita H. F. Chang,et al.  Acceleration carries the local inversion effect in biological motion perception. , 2009, Journal of vision.

[34]  Reinhard Blickhan,et al.  Compliant leg behaviour explains basic dynamics of walking and running , 2006, Proceedings of the Royal Society B: Biological Sciences.

[35]  Soon Ki Jung,et al.  Tracking Noisy Targets: A Review of Recent Object Tracking Approaches , 2018, ArXiv.

[36]  Patrice D. Tremoulet,et al.  Perceptual causality and animacy , 2000, Trends in Cognitive Sciences.

[37]  Pietro Perona,et al.  Caltech-UCSD Birds 200 , 2010 .

[38]  Dorita H. F. Chang,et al.  Body Configuration Modulates the Usage of Local Cues to Direction in Biological-Motion Perception , 2011, Psychological science.

[39]  C. Lintott,et al.  Snapshot Serengeti, high-frequency annotated camera trap images of 40 mammalian species in an African savanna , 2015, Scientific Data.

[40]  Nikolaus F. Troje,et al.  What is biological motion?: Definition, stimuli and paradigms , 2012 .

[41]  Tad McGeer,et al.  Passive Dynamic Walking , 1990, Int. J. Robotics Res..

[42]  N. Troje.  Decomposing biological motion: a framework for analysis and synthesis of human gait patterns. , 2002, Journal of vision.

[43]  D. Regan,et al.  Looming detectors in the human visual pathway , 1978, Vision Research.

[44]  P. Rousseeuw.  Silhouettes: a graphical aid to the interpretation and validation of cluster analysis , 1987 .

[45]  Pietro Perona,et al.  Recognition in Terra Incognita , 2018, ECCV.

[46]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[47]  Tony X. Han,et al.  Deep convolutional neural network based species recognition for wild animal monitoring , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[48]  Cordelia Schmid,et al.  Dense Trajectories and Motion Boundary Descriptors for Action Recognition , 2013, International Journal of Computer Vision.

[49]  Cordelia Schmid,et al.  Learning realistic human actions from movies , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[50]  Giorgio Vallortigara,et al.  Gravity bias in the interpretation of biological motion by inexperienced chicks , 2006, Current Biology.

[51]  Thomas S. Huang,et al.  Foreground object detection in highly dynamic scenes using saliency , 2014, 2014 IEEE International Conference on Image Processing (ICIP).

[52]  S. Lea,et al.  Perception of Emotion from Dynamic Point-Light Displays Represented in Dance , 1996, Perception.

[53]  D. Hu,et al.  Friction enhancement in concertina locomotion of snakes , 2012, Journal of The Royal Society Interface.

[54]  M. Labarbera.  Why the Wheels Won't Go , 1983, The American Naturalist.

[55]  Dorita H. F. Chang,et al.  Perception of animacy and direction from local biological motion signals. , 2008, Journal of vision.

[56]  Jitendra Malik,et al.  Object Segmentation by Long Term Analysis of Point Trajectories , 2010, ECCV.

[57]  H. Akaike,et al.  Information Theory and an Extension of the Maximum Likelihood Principle , 1973 .

[58]  Adrian Hilton,et al.  Computer vision for sports: Current applications and research topics , 2017, Comput. Vis. Image Underst..

[59]  Geoffrey P Bingham,et al.  Trajectory forms as a source of information about events , 2002, Perception & psychophysics.

[60]  Luc Van Gool,et al.  The 2019 DAVIS Challenge on VOS: Unsupervised Multi-Object Segmentation , 2019, ArXiv.

[61]  Adrian Hilton,et al.  Computer Vision in Sports , 2014, Advances in Computer Vision and Pattern Recognition.

[62]  Kevin Cannons,et al.  A Review of Visual Tracking , 2008 .

[63]  Marc H. Raibert,et al.  Legged Robots That Balance , 1986, IEEE Expert.

[64]  Jonathan M. Borwein,et al.  The Harpercollins Dictionary Of Mathematics , 1991 .

[65]  Cees Snoek,et al.  APT: Action localization proposals from dense trajectories , 2015, BMVC.

[66]  Mubarak Shah,et al.  Action MACH a spatio-temporal Maximum Average Correlation Height filter for action recognition , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[67]  Joseph Katz,et al.  Large amplitude unsteady motion of a flexible slender propulsor , 1979, Journal of Fluid Mechanics.

[68]  Nikolaus F. Troje,et al.  3D Periodic Human Motion Reconstruction from 2D Motion Sequences , 2007, Neural Computation.

[69]  Corinna Cortes,et al.  Support-Vector Networks , 1995, Machine Learning.

[70]  Тараса Шевченка,et al.  Quo vadis? , 2013, Clinical chemistry.

[71]  Russ Tedrake,et al.  Efficient Bipedal Robots Based on Passive-Dynamic Walkers , 2005, Science.

[72]  David J. Fleet,et al.  Physics-Based Person Tracking Using Simplified Lower-Body Dynamics , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[73]  Andrew Zisserman,et al.  Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[74]  T. Poggio,et al.  Hierarchical models of object recognition in cortex , 1999, Nature Neuroscience.

[75]  Thomas Mensink,et al.  Improving the Fisher Kernel for Large-Scale Image Classification , 2010, ECCV.

[76]  F. Heider,et al.  An experimental study of apparent behavior , 1944 .

[77]  Jiangping Wang,et al.  Automated identification of animal species in camera trap images , 2013, EURASIP J. Image Video Process..

[78]  Yong-Liang Yu,et al.  A flow control mechanism in wing flapping with stroke asymmetry during insect forward flight , 2005 .

[79]  Donald W. Bouldin,et al.  A Cluster Separation Measure , 1979, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[80]  A. M. Berg,et al.  Wing and body kinematics of takeoff and landing flight in the pigeon (Columba livia) , 2010, Journal of Experimental Biology.

[81]  T. McMahon,et al.  The mechanics of running: how does stiffness couple with speed? , 1990, Journal of biomechanics.

[82]  Christopher Hunt,et al.  Notes on the OpenSURF Library , 2009 .

[83]  Richard P. Wildes,et al.  Review of Action Recognition and Detection Methods , 2016, ArXiv.

[84]  M. Goodale,et al.  Separate visual pathways for perception and action , 1992, Trends in Neurosciences.

[85]  Xavier Binefa,et al.  Robust Real-Time Periodic Motion Detection, Analysis, and Applications , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[86]  Mubarak Shah,et al.  UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild , 2012, ArXiv.

[87]  Khurram Soomro,et al.  Chapter 9 Action Recognition in Realistic Sports Videos , 2015 .

[88]  Erik G. Learned-Miller,et al.  It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos , 2016, ECCV.

[89]  Jiri Matas,et al.  A Novel Performance Evaluation Methodology for Single-Target Trackers , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[90]  Vic Braden,et al.  Recognising the Style of Spatially Exaggerated Tennis Serves , 2001, Perception.

[91]  Mohieddine Jelali,et al.  Detection and Diagnosis of Stiction in Control Loops , 2010 .

[92]  Zhi Zhang,et al.  Coupled ensemble graph cuts and object verification for animal segmentation from highly cluttered videos , 2015, 2015 IEEE International Conference on Image Processing (ICIP).

[93]  J. C. Dunn,et al.  A Fuzzy Relative of the ISODATA Process and Its Use in Detecting Compact Well-Separated Clusters , 1973 .

[94]  T. Poggio,et al.  Biologically Plausible Neural Model for the Recognition of Biological Motion and Actions , 2002 .

[95]  A. Garrod.  Animal Locomotion , 1874, Nature.

[96]  R. Blickhan,et al.  Similarity in multilegged locomotion: Bouncing like a monopode , 1993, Journal of Comparative Physiology A.

[97]  Randal C. Nelson,et al.  Detection and Recognition of Periodic, Nonrigid Motion , 1997, International Journal of Computer Vision.

[98]  Guosheng Lin,et al.  Video Object Segmentation and Tracking: A Survey , 2019, ArXiv.

[99]  J. Cutting,et al.  Temporal and spatial factors in gait perception that influence gender recognition , 1978, Perception & psychophysics.

[100]  Geoffrey P. Bingham,et al.  Dynamics and the problem of visual event recognition , 1996 .

[101]  R. Blake.  Cats Perceive Biological Motion , 1993 .

[102]  T. McMahon.  The role of compliance in mammalian running gaits. , 1985, The Journal of experimental biology.

[103]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[104]  Takeo Kanade,et al.  Computer Vision and Image Understanding Computer Vision for Assistive Technologies , 2022 .

[105]  Marc H. Raibert,et al.  Running on four legs as though they were one , 1986, IEEE J. Robotics Autom..

[106]  Zhi Zhang,et al.  Animal Detection From Highly Cluttered Natural Scenes Using Spatiotemporal Object Region Proposals and Patch Verification , 2016, IEEE Transactions on Multimedia.

[107]  Yun Fu,et al.  Human Action Recognition and Prediction: A Survey , 2018, International Journal of Computer Vision.

[108]  G Johansson,et al.  Vector analysis in visual perception of rolling motion , 1974, Psychologische Forschung.

[109]  T. Poggio,et al.  Cognitive neuroscience: Neural mechanisms for the recognition of biological movements , 2003, Nature Reviews Neuroscience.

[110]  Jintao Li,et al.  Hierarchical spatio-temporal context modeling for action recognition , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[111]  Cordelia Schmid,et al.  Action recognition by dense trajectories , 2011, CVPR 2011.

[112]  Stephen Hailes,et al.  Upwash exploitation and downwash avoidance by flap phasing in ibis formation flight , 2014, Nature.

[113]  R. McNeill Alexander,et al.  Principles of Animal Locomotion , 2002 .

[114]  Andrew Zisserman,et al.  Convolutional Two-Stream Network Fusion for Video Action Recognition , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[115]  Steven M. Seitz,et al.  View-Invariant Analysis of Cyclic Motion , 1997, International Journal of Computer Vision.

[116]  R. Crowninshield,et al.  A physiologically based criterion of muscle force prediction in locomotion. , 1981, Journal of biomechanics.

[117]  Pietro Perona,et al.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[118]  D. D. Hoffman,et al.  The interpretation of biological motion , 1982, Biological Cybernetics.

[119]  Christian Hennig,et al.  Recovering the number of clusters in data sets with noise features using feature rescaling factors , 2015, Inf. Sci..

[120]  Christopher Joseph Pal,et al.  Activity recognition using the velocity histories of tracked keypoints , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[121]  Manoj Srinivasan,et al.  Computer optimization of a minimal biped model discovers walking and running , 2006, Nature.

[122]  Hilary Bart-Smith,et al.  Hydrodynamic Performance of Aquatic Flapping: Efficiency of Underwater Flight in the Manta , 2016 .

[123]  G. Johansson.  Visual perception of biological motion and a model for its analysis , 1973 .

[124]  M. Triantafyllou,et al.  Hydrodynamics of Fishlike Swimming , 2000 .

[125]  M Calisti,et al.  Fundamentals of soft robot locomotion , 2017, Journal of The Royal Society Interface.

[126]  Agnieszka C. Miguel,et al.  Finding areas of motion in camera trap images , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[127]  Cordelia Schmid,et al.  Human Detection Using Oriented Histograms of Flow and Appearance , 2006, ECCV.

[128]  Deva Ramanan,et al.  Exploring Weak Stabilization for Motion Feature Extraction , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[129]  Arnold W. M. Smeulders,et al.  Real-World Repetition Estimation by Div, Grad and Curl , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[130]  Terrance E. Boult,et al.  Animal recognition in the Mojave Desert: Vision tools for field biologists , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[131]  Yuji Ikegaya,et al.  Cortical and subcortical responses to biological motion , 2018, NeuroImage.