Recording behaviour of indoor-housed farm animals automatically using machine vision technology: A systematic review

Large-scale phenotyping of animal behaviour traits is time consuming and has led to increased demand for technologies that can automate these procedures. Automated tracking of animals has been successful in controlled laboratory settings, but recording from animals in large groups in highly variable farm settings presents challenges. The aim of this review is to provide a systematic overview of the advances that have occurred in automated, high throughput image detection of farm animal behavioural traits with welfare and production implications. Peer-reviewed publications written in English were reviewed systematically following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. After identification, screening, and assessment for eligibility, 108 publications met these specifications and were included for qualitative synthesis. Data collected from the papers included camera specifications, housing conditions, group size, algorithm details, procedures, and results. Most studies utilized standard digital colour video cameras for data collection, with increasing use of 3D cameras in papers published after 2013. Papers including pigs (across production stages) were the most common (n = 63). The most common behaviours recorded included activity level, area occupancy, aggression, gait scores, resource use, and posture. Our review revealed many overlaps in methods applied to analysing behaviour, and most studies started from scratch instead of building upon previous work. Training and validation sample sizes were generally small (mean±s.d. groups = 3.8±5.8) and in data collection and testing took place in relatively controlled environments. To advance our ability to automatically phenotype behaviour, future research should build upon existing knowledge and validate technology under commercial settings and publications should explicitly describe recording conditions in detail to allow studies to be reproduced.

[1]  M. Dawkins,et al.  Optical flow, flock behaviour and chicken welfare , 2012, Animal Behaviour.

[2]  Daniel Berckmans,et al.  Model-based calving monitor using real time image analysis , 2006 .

[3]  Murat Kulahci,et al.  Pig herd monitoring and undesirable tripping and stepping prevention , 2015, Comput. Electron. Agric..

[4]  Ilias Kyriazakis,et al.  Automated tracking to measure behavioural changes in pigs for health and welfare monitoring , 2017, Scientific Reports.

[5]  Paulo Aguiar,et al.  ThermoLabAnimal – A high-throughput analysis software for non-invasive thermal assessment of laboratory mice , 2019, Physiology & Behavior.

[6]  Zhu Weixing,et al.  Detection of porcine respiration based on machine vision , 2010, 2010 Third International Symposium on Knowledge Acquisition and Modeling.

[7]  Stefano Viazzi,et al.  Image feature extraction for classification of aggressive interactions among pigs , 2014 .

[8]  Ehsan Khoramshahi,et al.  Real-time recognition of sows in video: A supervised approach , 2014 .

[9]  Pedro Gonçalves,et al.  An IoT-Based Solution for Intelligent Farming † , 2019, Sensors.

[10]  路淳 The Animal Sound , 2009 .

[11]  Zhang Jin,et al.  Identification of abnormal gait of pigs based on video analysis , 2010, 2010 Third International Symposium on Knowledge Acquisition and Modeling.

[12]  Hongqian Chen,et al.  Evaluation of a laying-hen tracking algorithm based on a hybrid support vector machine , 2016, Journal of Animal Science and Biotechnology.

[13]  Irenilza de Alencar Nääs,et al.  Computational Vision Use for Evaluation of Confined Dairy Cows Behavior , 2008 .

[14]  D M Weary,et al.  Board-invited review: Using behavior to predict and identify ill health in animals. , 2009, Journal of animal science.

[15]  Torben Gregersen,et al.  Original papers: Development of a real-time computer vision system for tracking loose-housed pigs , 2011 .

[16]  Claudia Arcidiacono,et al.  A computer vision-based system for the automatic detection of lying behaviour of dairy cows in free-stall barns , 2013 .

[17]  Kanda Runapongsa Saikaew,et al.  Boundary Detection of Pigs in Pens Based on Adaptive Thresholding Using an Integral Image and Adaptive Partitioning , 2017 .

[18]  Sandra A. Edwards,et al.  Using automated image analysis in pig behavioural research: Assessment of the influence of enrichment substrate provision on lying behaviour , 2017 .

[19]  Wei Li,et al.  Validity of the Microsoft Kinect sensor for assessment of normal walking patterns in pigs , 2015, Comput. Electron. Agric..

[20]  Jinchang Ren,et al.  Automatic Animal Detection from Kinect Sensed Images for Livestock Monitoring and Assessment , 2015, 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing.

[21]  Kang-Sun Choi,et al.  Cow Behavior Recognition Using Motion History Image Feature , 2017, ICIAR.

[22]  B. Sturm,et al.  A new approach for categorizing pig lying behaviour based on a Delaunay triangulation method. , 2017, Animal : an international journal of animal bioscience.

[23]  Stefano Viazzi,et al.  Validation of a commercial system for the continuous and automated monitoring of dairy cows activity , 2015 .

[24]  S. Desire,et al.  Prediction of reduction in aggressive behaviour of growing pigs using skin lesion traits as selection criteria. , 2016, Animal : an international journal of animal bioscience.

[25]  Xavier Masip-Bruin,et al.  Smart Computing and Sensing Technologies for Animal Welfare , 2016, ACM Comput. Surv..

[26]  Michael Vinther,et al.  Validation of a digital video tracking system for recording pig locomotor behaviour , 2005, Journal of Neuroscience Methods.

[27]  Ephraim Maltz,et al.  Automatic lameness detection based on consecutive 3D-video recordings , 2014 .

[28]  Suresh Neethirajan,et al.  Recent advances in wearable sensors for animal health management , 2017 .

[29]  Daniel Berckmans,et al.  Automatic detection of lameness in dairy cattle-Vision-based trackway analysis in cow's locomotion , 2008 .

[30]  N Li,et al.  Review: Automated techniques for monitoring the behaviour and welfare of broilers and laying hens: towards the goal of precision livestock farming. , 2019, Animal : an international journal of animal bioscience.

[31]  Fernand Meyer,et al.  Topographic distance and watershed lines , 1994, Signal Process..

[32]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. , 2010, International journal of surgery.

[33]  Jørgen Kongsro Development of a computer vision system to monitor pig locomotion , 2013 .

[34]  Du-Ming Tsai,et al.  A motion and image analysis method for automatic detection of estrus and mating behavior in cattle , 2014 .

[35]  Yongwha Chung,et al.  Segmentation of group-housed pigs using concave points and edge information , 2017, 2017 19th International Conference on Advanced Communication Technology (ICACT).

[36]  Y. Chung SEGMENTATION METHODS FOR A GROUP-HOUSED PIG MONITORING SYSTEM , 2017 .

[37]  Hongwei Xin,et al.  Real-time Assessment of Swine Thermal Comfort by Computer Vision , 2002 .

[38]  Chen Chen,et al.  Image motion feature extraction for recognition of aggressive behaviors among group-housed pigs , 2017, Comput. Electron. Agric..

[39]  Hongwei Xin,et al.  Digital Repository @ Iowa State University Automated Tracking and Behavior Quantification of Laying Hens Using 3D Computer Vision and Radio Frequency Identification Technologies , 2022 .

[40]  Daniel Berckmans,et al.  Classification of aggressive behaviour in pigs by activity index and multilayer feed forward neural network , 2014 .

[41]  Claudia Arcidiacono,et al.  The automatic detection of dairy cow feeding and standing behaviours in free-stall barns by a computer vision-based system , 2015 .

[42]  M. Dawkins,et al.  Using behaviour to assess animal welfare , 2004, Animal Welfare.

[43]  D. Berckmans,et al.  Precision livestock farming technologies for welfare management in intensive livestock systems. , 2014, Revue scientifique et technique.

[44]  Yongwha Chung,et al.  Automatic Recognition of Aggressive Behavior in Pigs Using a Kinect Depth Sensor , 2016, Sensors.

[45]  Cécile Cornou,et al.  Automatic detection of deviations in activity levels in groups of broiler chickens – A pilot study , 2011 .

[46]  A. Lawrence,et al.  Pigs' aggressive temperament affects pre-slaughter mixing aggression, stress and meat quality. , 2010, Animal : an international journal of animal bioscience.

[47]  Christer Bergsten,et al.  Learning Based Image Segmentation of Pigs in a Pen , 2014 .

[48]  Vasileios Exadaktylos,et al.  Towards real-time control of chicken activity in a ventilated chamber , 2015 .

[49]  T. Leroy,et al.  Automatic real-time monitoring of locomotion and posture behaviour of pregnant cows prior to calving using online image analysis , 2008 .

[50]  Melvyn L. Smith,et al.  Towards on-farm pig face recognition using convolutional neural networks , 2018, Comput. Ind..

[51]  Weixing Zhu,et al.  Pig target extraction based on adaptive elliptic block and wavelet edge detection , 2016, ICSPS 2016.

[52]  Ikuo Kobayashi,et al.  A General Video Surveillance Framework for Animal Behavior Analysis , 2016, 2016 Third International Conference on Computing Measurement Control and Sensor Network (CMCSN).

[53]  Aaron Ingham,et al.  Cattle behaviour classification from collar, halter, and ear tag sensors , 2017 .

[54]  Daniel Berckmans,et al.  The automatic monitoring of pigs water use by cameras , 2013 .

[55]  Alain N. Rousseau,et al.  Rethinking environment control strategy of confined animal housing systems through precision livestock farming , 2017 .

[56]  M. Dawkins,et al.  Optical flow patterns in broiler chicken flocks as automated measures of behaviour and gait , 2009 .

[57]  Murat Kulahci,et al.  Monitoring pig movement at the slaughterhouse using optical flow and modified angular histograms , 2016 .

[58]  Daniel Berckmans,et al.  The use of image analysis as a new approach to assess behaviour classification in a pig barn , 2013 .

[59]  Ilan Halachmi,et al.  3D Computer-vision system for automatically estimating heifer height and body mass , 2017, Biosystems Engineering.

[60]  Lance C. Pérez,et al.  Health Monitoring of Group-Housed Pigs using Depth-Enabled Multi-Object Tracking , 2016 .

[61]  D. Berckmans,et al.  Review: Precision livestock farming: building 'digital representations' to bring the animals closer to the farmer. , 2019, Animal : an international journal of animal bioscience.

[62]  M. Špinka,et al.  Computer-aided method for calculating animal configurations during social interactions from two-dimensional coordinates of color-marked body parts , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[63]  N. Otsu A threshold selection method from gray level histograms , 1979 .

[64]  Lene Juul Pedersen,et al.  Segmentation of sows in farrowing pens , 2014, IET Image Process..

[65]  H Xin,et al.  Assessing swine thermal comfort by image analysis of postural behaviors. , 1999, Journal of animal science.

[66]  Xunmu Zhu,et al.  Automatic recognition of lactating sow postures from depth images by deep learning detector , 2018, Comput. Electron. Agric..

[67]  Gong Zhang,et al.  Separation of Touching Grain Kernels in an Image by Ellipse Fitting Algorithm , 2005 .

[68]  Dries Berckmans,et al.  Developing precision livestock farming tools for precision dairy farming , 2017 .

[69]  Y. Le Cozler,et al.  High-precision scanning system for complete 3D cow body shape imaging and analysis of morphological traits , 2019, Comput. Electron. Agric..

[70]  S. Turner Breeding against harmful social behaviours in pigs and chickens: state of the art and the way forward , 2011 .

[71]  Daniel Berckmans,et al.  Comparison of a three-dimensional and two-dimensional camera system for automated measurement of back posture in dairy cows , 2014 .

[72]  P. Weintraub The Importance of Publishing Negative Results , 2016, Journal of insect science.

[73]  Daniel Berckmans,et al.  Automated video analysis of pig activity at pen level highly correlates to human observations of behavioural activities , 2014 .

[74]  Claudia Bahr,et al.  Original paper: Real-time automatic lameness detection based on back posture extraction in dairy cattle: Shape analysis of cow with image processing techniques , 2010 .

[75]  Vasileios Exadaktylos,et al.  Analysis of behavioural patterns in broilers using camera-based technology , 2015 .

[76]  Daniel Berckmans,et al.  Performance of an image analysis processing system for hen tracking in an environmental preference chamber. , 2014, Poultry science.

[77]  Daniel L. Schmoldt,et al.  Expert systems in forestry: Utilizing information and expertise for decision making☆ , 1986 .

[78]  Tse-Min Lee,et al.  Insecticidal Activity and Insect Repellency of Four Species of Sea Lily (Comatulida: Comatulidae) From Taiwan , 2016, Journal of insect science.

[79]  V. Gómez,et al.  An automatic colour-based computer vision algorithm for tracking the position of piglets , 2009 .

[80]  Yongwha Chung,et al.  An index algorithm for tracking pigs in pigsty , 2014 .

[81]  L P Noldus,et al.  EthoVision: A versatile video tracking system for automation of behavioral experiments , 2001, Behavior research methods, instruments, & computers : a journal of the Psychonomic Society, Inc.

[82]  John S. Zelek,et al.  Real-time automated concurrent visual tracking of many animals and subsequent behavioural compilation , 2001, Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001.

[83]  Elfed Lewis,et al.  FPGA based Real time 'secure' body temperature monitoring suitable for WBSN 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing , 2015 .

[84]  Daniel Berckmans,et al.  Automatic cough detection for bovine respiratory disease in a calf house , 2018, Biosystems Engineering.

[85]  Weixing Zhu,et al.  Identification of group-housed pigs based on Gabor and Local Binary Pattern features , 2018 .

[86]  Petra Perner,et al.  Motion Tracking of Animals for Behavior Analysis , 2001, IWVF.

[87]  Jay D. Harmon,et al.  Neural Network Analysis of Postural Behavior of Young Swine to Determine the IR Thermal Comfort State , 1997 .

[88]  Thomas Banhazi,et al.  Precision Livestock Farming: Precision feeding technologies and sustainable livestock production , 2012 .

[89]  Weixing Zhu,et al.  Foreground detection of group-housed pigs based on the combination of Mixture of Gaussians using prediction mechanism and threshold segmentation , 2014 .

[90]  S. Kestin,et al.  Prevalence of leg weakness in broiler chickens and its relationship with genotype , 1992, Veterinary Record.

[91]  A. Aydin,et al.  Using 3D vision camera system to automatically assess the level of inactivity in broiler chickens , 2017, Comput. Electron. Agric..

[92]  Tiemin Zhang,et al.  Development of an early warning algorithm to detect sick broilers , 2018, Comput. Electron. Agric..

[93]  D Berckmans,et al.  Automatic measurement of touch and release angles of the fetlock joint for lameness detection in dairy cattle using vision techniques. , 2012, Journal of dairy science.

[94]  Kenji Tsukamoto,et al.  Poultry tracking system with camera using particle filters , 2009, 2008 IEEE International Conference on Robotics and Biomimetics.

[95]  Andrea Pezzuolo,et al.  A Feasibility Study on the Use of a Structured Light Depth-Camera for Three-Dimensional Body Measurements of Dairy Cows in Free-Stall Barns , 2018, Sensors.

[96]  J. Houdmont,et al.  Application of multiple behaviour change models to identify determinants of farmers' biosecurity attitudes and behaviours. , 2018, Preventive veterinary medicine.

[97]  Qiming Zhu,et al.  Automatic early warning of tail biting in pigs: 3D cameras can detect lowered tail posture before an outbreak , 2018, PloS one.

[98]  Nigel J. B. McFarlane,et al.  Segmentation and tracking of piglets in images , 1995, Machine Vision and Applications.

[99]  Daniel Berckmans,et al.  Animal Sound...Talks! Real-time Sound Analysis for Health Monitoring in Livestock , 2015 .

[100]  Dries Berckmans,et al.  General introduction to precision livestock farming , 2017 .

[101]  Greg M. Cronin,et al.  Using video image analysis to count hens in cages and reduce egg breakage on collection belts , 2008 .

[102]  Y. Le Cozler,et al.  Volume and surface area of Holstein dairy cows calculated from complete 3D shapes acquired using a high-precision scanning system: Interest for body weight estimation , 2019, Comput. Electron. Agric..

[103]  Harold W. Kuhn,et al.  The Hungarian method for the assignment problem , 1955, 50 Years of Integer Programming.

[104]  Hongwei Xin,et al.  A real-time computer vision assessment and control of thermal comfort for group-housed pigs , 2008 .

[105]  P ? ? ? ? ? ? ? % ? ? ? ? , 1991 .

[106]  Yan Zhu,et al.  The Posture Recognition of Pigs Based on Zernike Moments and Support Vector Machines , 2015, 2015 10th International Conference on Intelligent Systems and Knowledge Engineering (ISKE).

[107]  U. Lindenberger,et al.  Historical trends in modifiable indicators of cardiovascular health and self-rated health among older adults: Cohort differences over 20 years between the Berlin Aging Study (BASE) and the Berlin Aging Study II (BASE-II) , 2018, PloS one.

[108]  Yongwha Chung,et al.  Touching-Pigs Segmentation using Concave Points in Continuous Video Frames , 2017, ICACS '17.

[109]  Mark Trotter,et al.  The role of interoperable data standards in precision livestock farming in extensive livestock systems: A review , 2019, Comput. Electron. Agric..

[110]  Daniel Berckmans,et al.  A COMPUTER VISION METHOD FOR ON-LINE BEHAVIORAL QUANTIFICATION OF INDIVIDUALLY CAGED POULTRY , 2006 .

[111]  M. Guarino,et al.  Technical note: Validation of a commercial system for the continuous and automated monitoring of dairy cow activity. , 2016, Journal of dairy science.

[112]  M Nilsson,et al.  Development of automatic surveillance of animal behaviour and welfare using image analysis and machine learned segmentation technique. , 2015, Animal : an international journal of animal bioscience.

[113]  B. Martínez-López,et al.  Motion-based video monitoring for early detection of livestock diseases: The case of African swine fever , 2017, PloS one.

[114]  Daniel Berckmans,et al.  Image-processing technique to measure pig activity in response to climatic variation in a pig barn , 2014 .

[115]  Sabine G. Gebhardt-Henrich,et al.  A Systematic Review of Precision Livestock Farming in the Poultry Sector: Is Technology Focussed on Improving Bird Welfare? , 2019, Animals : an open access journal from MDPI.

[116]  Yongwha Chung,et al.  Automatic Detection and Recognition of Pig Wasting Diseases Using Sound Data in Audio Surveillance Systems , 2013, Sensors.

[117]  D. Berckmans,et al.  Application of a fully automatic analysis tool to assess the activity of broiler chickens with different gait scores , 2010 .

[118]  Sandra A. Edwards,et al.  Automatic detection of mounting behaviours among pigs using image analysis , 2016, Comput. Electron. Agric..

[119]  Daniel Berckmans,et al.  Automatic Identification of Activity and Spatial Use of Broiler Chickens with Different Gait Scores , 2013 .

[120]  Weixing Zhu,et al.  Recognition and drinking behaviour analysis of individual pigs based on machine vision , 2017 .

[121]  Kai Liu,et al.  Automatic recognition of lactating sow behaviors through depth image processing , 2016, Comput. Electron. Agric..

[122]  V. Alchanatis,et al.  Comparison of segmentation algorithms for cow contour extraction from natural barn background in side view images , 2013 .

[123]  Thomas Nickson,et al.  Monitoring chicken flock behaviour provides early warning of infection by human pathogen Campylobacter , 2016, Proceedings of the Royal Society B: Biological Sciences.

[124]  Daniel Berckmans,et al.  Automatic Monitoring of Pig Activity Using Image Analysis , 2013, ACIVS.

[125]  V Goedseels,et al.  Image analysis to measure activity index of animals. , 2010, Equine veterinary journal. Supplement.

[126]  Karl Johan Åström,et al.  5.6. Continuous surveillance of pigs in a pen using learning-based segmentation in computer vision , 2015 .

[127]  Daniel Berckmans,et al.  Automatic Identification of Marked Pigs in a Pen Using Image Pattern Recognition , 2013, MDA.

[128]  S. Edwards,et al.  The ‘Real Welfare’ scheme: benchmarking welfare outcomes for commercially farmed pigs , 2017, Animal : an international journal of animal bioscience.

[129]  C. Lokhorst,et al.  On farm implementation of a fully automatic computer vision system for monitoring gait related measures in dairy cows. , 2013 .

[130]  D. Sergeant,et al.  Computer visual tracking of poultry , 1998 .

[131]  Victor A. Kulikov,et al.  Application of 3-D imaging sensor for tracking minipigs in the open field test , 2014, Journal of Neuroscience Methods.

[132]  Melvyn L. Smith,et al.  Automated monitoring of dairy cow body condition, mobility and weight using a single 3D video capture device , 2018, Comput. Ind..

[133]  Xia Sun,et al.  State-of-the-Art Internet of Things in Protected Agriculture , 2019, Sensors.

[134]  D. Stajnko,et al.  Estimation of bull live weight through thermographically measured body dimensions , 2008 .

[135]  Daniel Berckmans,et al.  Development of an early warning system for a broiler house using computer vision , 2013 .

[136]  Xincheng Li,et al.  Automated detection of sick pigs based on machine vision , 2009, 2009 IEEE International Conference on Intelligent Computing and Intelligent Systems.

[137]  Jay D. Harmon,et al.  Comparison of image feature extraction for classification of swine thermal comfort behavior , 1998 .

[138]  Richard S Gates,et al.  Machine vision to identify broiler breeder behavior , 2013 .

[139]  Daniel Berckmans,et al.  Appropriate data visualisation is key to Precision Livestock Farming acceptance , 2017, Comput. Electron. Agric..

[140]  Yongwha Chung,et al.  Lying-Pig Detection using Depth Information , 2017, ICACS '17.

[141]  J. Sánchez-Vizcaíno,et al.  Early Detection of Infection in Pigs through an Online Monitoring System , 2017, Transboundary and emerging diseases.

[142]  D. Berckmans,et al.  Computer vision based recognition of behavior phenotypes of laying hens , 2005 .

[143]  F. Petrera,et al.  A Survey of Italian Dairy Farmers’ Propensity for Precision Livestock Farming Tools , 2019, Animals : an open access journal from MDPI.

[144]  Yongwha Chung,et al.  A Cost-Effective Pigsty Monitoring System Based on a Video Sensor , 2014, KSII Trans. Internet Inf. Syst..

[145]  Daniel Berckmans,et al.  Automatic estimation of number of piglets in a pen during farrowing, using image analysis , 2016 .

[146]  David V. Anderson,et al.  Identifying rale sounds in chickens using audio signals for early disease detection in poultry , 2016, 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP).

[147]  Weixing Zhu,et al.  Multi-object extraction from topview group-housed pig images based on adaptive partitioning and multilevel thresholding segmentation , 2015 .

[148]  Yongwha Chung,et al.  Depth-Based Detection of Standing-Pigs in Moving Noise Environments , 2017, Sensors.

[149]  D Berckmans,et al.  Analysis of individual classification of lameness using automatic measurement of back posture in dairy cattle. , 2013, Journal of dairy science.