A CNN-based posture change detection for lactating sow in untrimmed depth videos

Abstract The dangerous body movements of lactating sows are mainly reflected in the frequency and duration of posture changes. To avoid the disturbances from the light environment and illumination variations caused by heat lamp and day-night cycle, depth videos were acquired for posture change detection. In this paper, a CNN-based method was developed to automatically detect sow posture change in untrimmed depth videos. Firstly, clip-level Detect and Classify Posture R-CNN (DCP R-CNN) was proposed for sow detection and posture classification. The DCP R-CNN decoupled the detection and classification task into two branches: detection branch and classification branch so that the detection branch could focus on the task of sow localization. Secondly, a method was proposed to temporally localize the possible posture changing segments from coarse to fine. Finally, a two-stream posture change classification network, which used two CNNs to extract the complementary features from DMM and the first and the last frames, was introduced to classify the possible posture changing segments into change or non-change. At the scene of commercial piggery, the videos were acquired at 5 frames per second (fps) by installing a Microsoft Kinect v2.0 sensor above the pen. The test set consisted of two 24-hour untrimmed depth videos and six 40-minute untrimmed depth videos. The test results showed that our method could detect the posture change correctly. The success rate of DCP R-CNN was 96.38% when the IoU threshold was 0.7, and the average precision and recall rate of 4 types of posture were 97.72% and 95.23%, respectively. The possible posture change was detected with a recall rate of 92.46% and the posture change was classified with an accuracy of 97.71%. The precision of posture change detection was 88.75%, the recall rate was 90.87%. The detection speed of our method was 5.71 fps, which was faster than the video frame rate (5 fps). Our study demonstrated the feasibility of 24-hour detection method for sow posture change.

[1]  Shuang Wang,et al.  RGB-D Hand-Held Object Recognition Based on Heterogeneous Feature Fusion , 2015, Journal of Computer Science and Technology.

[2]  Guy Lapalme,et al.  A systematic analysis of performance measures for classification tasks , 2009, Inf. Process. Manag..

[3]  Andrea Vedaldi,et al.  Understanding deep image representations by inverting them , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Cécile Cornou,et al.  Monitoring individual activity before, during and after parturition using sensors for sows with and without straw amendment , 2014 .

[5]  Aqing Yang,et al.  Automatic recognition of sow nursing behaviour using deep learning-based segmentation and spatial and temporal features , 2018, Biosystems Engineering.

[6]  Kai Liu,et al.  Automatic recognition of lactating sow behaviors through depth image processing , 2016, Comput. Electron. Agric..

[7]  Rongrong Ji,et al.  RGBD Salient Object Detection: A Benchmark and Algorithms , 2014, ECCV.

[8]  Lene Juul Pedersen,et al.  Nest building and posture changes and activity budget of gilts housed in pens and crates , 2014 .

[9]  Suman Saha,et al.  Deep Learning for Detecting Multiple Space-Time Action Tubes in Videos , 2016, BMVC.

[10]  Yuning Jiang,et al.  Acquisition of Localization Confidence for Accurate Object Detection , 2018, ECCV.

[11]  Donald M. Broom,et al.  The influence of sow behaviour on piglet mortality due to crushing in an open farrowing system , 2001 .

[12]  Yibin Li,et al.  Region-sequence based six-stream CNN features for general and fine-grained human action recognition in videos , 2018, Pattern Recognit..

[13]  Shiquan Cui,et al.  A comparison of postural changes and maternal responsiveness during early lactation in Landrace and Minpig sows , 2011 .

[14]  David Fraser,et al.  Sow body movements that crush piglets: a comparison between two types of farrowing accommodation , 1996 .

[15]  Shih-Fu Chang,et al.  CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[16]  Yu Qiao,et al.  Action Recognition with Stacked Fisher Vectors , 2014, ECCV.

[17]  J.K. Aggarwal,et al.  Human activity analysis , 2011, ACM Comput. Surv..

[18]  Yi Wu,et al.  Online Object Tracking: A Benchmark , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[20]  Murat Kulahci,et al.  Monitoring pig movement at the slaughterhouse using optical flow and modified angular histograms , 2016 .

[21]  Anthony Widjaja,et al.  Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond , 2003, IEEE Transactions on Neural Networks.

[22]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[23]  C. F. Hansen,et al.  Confinement of lactating sows in crates for 4 days after farrowing reduces piglet mortality. , 2013, Animal : an international journal of animal bioscience.

[24]  Ming Yang,et al.  BSN: Boundary Sensitive Network for Temporal Action Proposal Generation , 2018, ECCV.

[25]  Alexander J. Smola,et al.  Learning with Kernels: support vector machines, regularization, optimization, and beyond , 2001, Adaptive computation and machine learning series.

[26]  Daniel Berckmans,et al.  Classification of aggressive behaviour in pigs by activity index and multilayer feed forward neural network , 2014 .

[27]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Jitendra Malik,et al.  Finding action tubes , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Ilias Kyriazakis,et al.  Porcine lie detectors: Automatic quantification of posture state and transitions in sows using inertial sensors , 2016, Comput. Electron. Agric..

[30]  Stephen V. Stehman,et al.  Selecting and interpreting measures of thematic classification accuracy , 1997 .

[31]  M. Young,et al.  Predicting Parturition Time through Ultrasonic Measurement of Posture Changing Rate in Crated Landrace Sows , 2007 .

[32]  Xiaodong Yang,et al.  Recognizing actions using depth motion maps-based histograms of oriented gradients , 2012, ACM Multimedia.

[33]  Xunmu Zhu,et al.  Automatic recognition of lactating sow postures from depth images by deep learning detector , 2018, Comput. Electron. Agric..

[34]  Mario Cannataro,et al.  Protein-to-protein interactions: Technologies, databases, and algorithms , 2010, CSUR.

[35]  Rfm Oliveira,et al.  Behaviour and performance of lactating sows housed in different types of farrowing rooms during summer , 2011 .

[36]  Ramakant Nevatia,et al.  TALL: Temporal Activity Localization via Language Query , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[37]  Yutaka Satoh,et al.  Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[38]  Lorenzo Torresani,et al.  Learning Spatiotemporal Features with 3D Convolutional Networks , 2014, 2015 IEEE International Conference on Computer Vision (ICCV).

[39]  Lorenzo Torresani,et al.  Detect-and-Track: Efficient Pose Estimation in Videos , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.