Separating the Structural Components of Maize for Field Phenotyping Using Terrestrial LiDAR Data and Deep Convolutional Neural Networks

Separating structural components is important but also challenging for plant phenotyping and precision agriculture. Light detection and ranging (LiDAR) technology can potentially overcome these difficulties by providing high quality data. However, there are difficulties in automatically classifying and segmenting components of interest. Deep learning can extract complex features, but it is mostly used with images. Here, we propose a voxel-based convolutional neural network (VCNN) for maize stem and leaf classification and segmentation. Maize plants at three different growth stages were scanned with a terrestrial LiDAR and the voxelized LiDAR data were used as inputs. A total of 3000 individual plants (22 004 leaves and 3000 stems) were prepared for training through data augmentation, and 103 maize plants were used to evaluate the accuracy of classification and segmentation at both instance and point levels. The VCNN was compared with traditional clustering methods ( $K$ -means and density-based spatial clustering of applications with noise), a geometry-based segmentation method, and state-of-the-art deep learning methods (PointNet and PointNet++). The results showed that: 1) at the instance level, the mean accuracy of classification and segmentation (F-score) were 1.00 and 0.96, respectively; 2) at the point level, the mean accuracy of classification and segmentation (F-score) were 0.91 and 0.89, respectively; 3) the VCNN method outperformed traditional clustering methods; and 4) the VCNN was on par with PointNet and PointNet++ in classification, and performed the best in segmentation. The proposed method demonstrated LiDAR’s ability to separate structural components for crop phenotyping using deep learning, which can be useful for other fields.

[1]  W. Cohen,et al.  Lidar Remote Sensing for Ecosystem Studies , 2002 .

[2]  K. Zhao,et al.  Retrieving the gap fraction, element clumping index, and leaf area index of individual trees using single-scan data from a terrestrial laser scanner , 2017 .

[3]  Qian Qian,et al.  Regulation of OsSPL14 by OsmiR156 defines ideal plant architecture in rice , 2010, Nature Genetics.

[4]  Maryam Rahnemoonfar,et al.  Deep Count: Fruit Counting Based on Deep Simulated Learning , 2017, Sensors.

[5]  Shiming Xiang,et al.  3D Point Cloud Classification Based on Discrete Conditional Random Field , 2017, Edutainment.

[6]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[7]  Darko Stefanovic,et al.  Deep Neural Networks Based Recognition of Plant Diseases by Leaf Image Classification , 2016, Comput. Intell. Neurosci..

[8]  Shie Mannor,et al.  A Tutorial on the Cross-Entropy Method , 2005, Ann. Oper. Res..

[9]  Éric Gaussier,et al.  A Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation , 2005, ECIR.

[10]  K. Omasa,et al.  Estimation of vertical plant area density profiles in a rice canopy at different growth stages by high-resolution portable scanning lidar with a lightweight mirror , 2012 .

[11]  Jun Xue,et al.  Effects of light intensity within the canopy on maize lodging , 2016 .

[12]  M. Liu,et al.  A high-throughput stereo-imaging system for quantifying rape leaf traits during the seedling stage , 2017, Plant Methods.

[13]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Sebastian Scherer,et al.  3D Convolutional Neural Networks for landing zone detection from LiDAR , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[15]  Kilian Q. Weinberger,et al.  Distance Metric Learning for Large Margin Nearest Neighbor Classification , 2005, NIPS.

[16]  Chaoyang Wu,et al.  Airborne LiDAR technique for estimating biomass components of maize: A case study in Zhangye City, Northwest China , 2015 .

[17]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[18]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[19]  Gernot Riegler,et al.  OctNet: Learning Deep 3D Representations at High Resolutions , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Q. Guo,et al.  A geometric method for wood-leaf separation using terrestrial and simulated Lidar data , 2015 .

[21]  F. Baret,et al.  High-Throughput Phenotyping of Plant Height: Comparing Unmanned Aerial Vehicles and Ground LiDAR Estimates , 2017, Front. Plant Sci..

[22]  Shu Liu,et al.  Associatively Segmenting Instances and Semantics in Point Clouds , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]  Laura E. Chasmer,et al.  Filtering Stems and Branches from Terrestrial Laser Scanning Point Clouds Using Deep 3-D Fully Convolutional Networks , 2018, Remote. Sens..

[24]  Mohammad Najafi,et al.  Deep phenotyping: deep learning for temporal phenotype/genotype classification , 2017, Plant Methods.

[25]  N. A. Amusa,et al.  The Important Food Crops and Medicinal Plants of North-western Nigeria , 2005 .

[26]  Yin Zhou,et al.  VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[27]  Max Welling,et al.  Semi-Supervised Classification with Graph Convolutional Networks , 2016, ICLR.

[28]  Changying Li,et al.  In-field High Throughput Phenotyping and Cotton Plant Growth Analysis Using LiDAR , 2018, Front. Plant Sci..

[29]  Danil V. Prokhorov,et al.  A Convolutional Learning System for Object Classification in 3-D Lidar Data , 2010, IEEE Transactions on Neural Networks.

[30]  Peng Li,et al.  Segmenting tree crowns from terrestrial and mobile LiDAR data by exploring ecological theories , 2015 .

[31]  Hanno Scharr,et al.  Image Analysis: The New Bottleneck in Plant Phenotyping [Applications Corner] , 2015, IEEE Signal Processing Magazine.

[32]  Jörn Ostermann,et al.  A Crop/Weed Field Image Dataset for the Evaluation of Computer Vision Based Precision Agriculture Tasks , 2014, ECCV Workshops.

[33]  Seyed-Ahmad Ahmadi,et al.  V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation , 2016, 2016 Fourth International Conference on 3D Vision (3DV).

[34]  Yixin Chen,et al.  An End-to-End Deep Learning Architecture for Graph Classification , 2018, AAAI.

[35]  Dan Wu,et al.  Panicle-SEG: a robust image segmentation method for rice panicles in the field based on deep learning and superpixel optimization , 2017, Plant Methods.

[36]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[37]  Luca Maria Gambardella,et al.  Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence Flexible, High Performance Convolutional Neural Networks for Image Classification , 2022 .

[38]  Duc Thanh Nguyen,et al.  JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Pierre Vandergheynst,et al.  Geodesic Convolutional Neural Networks on Riemannian Manifolds , 2015, 2015 IEEE International Conference on Computer Vision Workshop (ICCVW).

[41]  R. MacCurdy,et al.  Three-Dimensional Root Phenotyping with a Novel Imaging and Software Platform1[C][W][OA] , 2011, Plant Physiology.

[42]  Leonidas J. Guibas,et al.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space , 2017, NIPS.

[43]  Rachid Guerraoui,et al.  Deep Learning Works in Practice. But Does it Work in Theory? , 2018, ArXiv.

[44]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[45]  Alan H. Strahler,et al.  Three-dimensional forest reconstruction and structural parameter retrievals using a terrestrial full-waveform lidar instrument (Echidna®) , 2013 .

[46]  Alan Wee-Chung Liew,et al.  Conditional Random Field and Deep Feature Learning for Hyperspectral Image Segmentation , 2017, ArXiv.

[47]  Volkan Isler,et al.  A comparative study of fruit detection and counting methods for yield mapping in apple orchards , 2018, J. Field Robotics.

[48]  Gaël Varoquaux,et al.  Scikit-learn: Machine Learning in Python , 2011, J. Mach. Learn. Res..

[49]  Shengli Tao,et al.  Perspectives and prospects of LiDAR in forest ecosystem monitoring and modeling , 2014 .

[50]  Vibhav Vineet,et al.  Conditional Random Fields as Recurrent Neural Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[51]  Jun Zhou,et al.  Conditional Random Field and Deep Feature Learning for Hyperspectral Image Classification , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[52]  Peter Langridge,et al.  Phenotyping approaches for physiological breeding and gene discovery in wheat , 2009 .

[53]  Baskar Ganapathysubramanian,et al.  An explainable deep machine vision framework for plant stress phenotyping , 2018, Proceedings of the National Academy of Sciences.

[54]  Sebastian Ruder,et al.  An Overview of Multi-Task Learning in Deep Neural Networks , 2017, ArXiv.

[55]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[56]  Luc Van Gool,et al.  Semantic Instance Segmentation with a Discriminative Loss Function , 2017, ArXiv.

[57]  Q. Guo,et al.  A bottom-up approach to segment individual deciduous trees using leaf-off lidar point cloud data , 2014 .

[58]  D. Ehlert,et al.  Measuring crop biomass density by laser triangulation , 2008 .

[59]  Yang Liu,et al.  O-CNN , 2017, ACM Trans. Graph..

[60]  James Patrick Underwood,et al.  Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards , 2016, J. Field Robotics.

[61]  Jose A. Jiménez-Berni,et al.  High Throughput Determination of Plant Height, Ground Cover, and Above-Ground Biomass in Wheat with LiDAR , 2018, Front. Plant Sci..

[62]  Kurt Keutzer,et al.  SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[63]  Patrick Wieschollek,et al.  Flex-Convolution - Million-Scale Point-Cloud Learning Beyond Grid-Worlds , 2018, ACCV.

[64]  Chuang Gan,et al.  End-to-End Learning of Motion Representation for Video Understanding , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[65]  Jin Liu,et al.  Stem–Leaf Segmentation and Phenotypic Trait Extraction of Individual Maize Using Terrestrial LiDAR Data , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[66]  Shang Gao,et al.  Deep Learning: Individual Maize Segmentation From Terrestrial Lidar Data Using Faster R-CNN and Regional Growth Algorithms , 2018, Front. Plant Sci..

[67]  Xin Zhang,et al.  End to End Learning for Self-Driving Cars , 2016, ArXiv.

[68]  Horst Bischof,et al.  OctNetFusion: Learning Depth Fusion from Data , 2017, 2017 International Conference on 3D Vision (3DV).

[69]  Hans-Peter Kriegel,et al.  A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise , 1996, KDD.

[70]  Jian Sun,et al.  Instance-Aware Semantic Segmentation via Multi-task Network Cascades , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[71]  Kerry Cawse-Nicholson,et al.  3D Tree Reconstruction from Simulated Small Footprint Waveform Lidar , 2013 .

[72]  J. MacQueen.  Some methods for classification and analysis of multivariate observations , 1967 .

[73]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[74]  Max Tegmark,et al.  Why Does Deep and Cheap Learning Work So Well? , 2016, Journal of Statistical Physics.

[75]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[76]  Subhransu Maji,et al.  Multi-view Convolutional Neural Networks for 3D Shape Recognition , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[77]  Lucas Beyer,et al.  In Defense of the Triplet Loss for Person Re-Identification , 2017, ArXiv.

[78]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[79]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[80]  Ashutosh Kumar Singh,et al.  Deep Learning for Plant Stress Phenotyping: Trends and Future Perspectives. , 2018, Trends in plant science.

[81]  Cheng Wang,et al.  Utility of multitemporal lidar for forest and carbon monitoring: Tree growth, biomass dynamics, and carbon flux , 2018 .

[82]  Q. Guo,et al.  Evaluating maize phenotype dynamics under drought stress using terrestrial lidar , 2019, Plant Methods.