PhenoStereo: a high-throughput stereo vision system for field-based plant phenotyping - with an application in sorghum stem diameter estimation

In recent years, three-dimensional (3D) sensing has gained a great interest in plant phenotyping because it can represent the 3D nature of plant architecture. Among all available 3D imaging technologies, stereo vision offers a viable solution due to its high spatial resolution and wide selection of camera modules. However, the performance of in-field stereo imaging for plant phenotyping has been adversely affected by textureless regions and occlusions of plants, and variable outdoor lighting and wind conditions. In this research, a portable stereo imaging module namely PhenoStereo was developed for high-throughput fieldbased plant phenotyping. PhenoStereo featured a self-contained embedded design, which made it capable of capturing images at 14 stereoscopic frames per second. In addition, a set of customized strobe lights was integrated to overcome lighting variations and enable the use of high shutter speed to overcome motion blurs. The stem diameter of sorghum plants is an important trait for stalk strength and biomass potential evaluation but has been identified as a challenging sensing task to automated in the field due to the complexity of the imaging object and the environment. To that connection, PhenoStereo was used to acquire a set of sorghum plant images and an automated point cloud data processing pipeline was also developed to automatically extract the stems and then quantify their diameters via an optimized 3D modeling process. The pipeline employed a Mask R-CNN deep learning network for detecting stalk contours and a Semi-Global Block Matching stereo matching algorithm for generating disparity maps. The correlation coefficient (r) between the image-derived stem diameters and the ground truth was 0.97 with a mean absolute error (MAE) of 1.44 mm, which outperformed any previously reported sensing approaches. These results demonstrated that with proper customization stereo vision can be a highly desirable sensing method for field-based plant phenotyping using high-fidelity 3D models reconstructed from stereoscopic images. With the proving results from sorghum plant stem diameter sensing, this proposed stereo sensing approach can likely be extended to characterize a broad spectrum of plant phenotypes such as leaf angle and tassel shape of maize plants and seed pods and stem nodes of soybean plants.

[1]  Shiming Xiang,et al.  Segment Graph Based Image Filtering: Fast Structure-Preserving Smoothing , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[2]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[3]  Ji Li,et al.  Crop recognition under weedy conditions based on 3 D imaging for robotic weed control , 2019 .

[4]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[5]  Jing Pang,et al.  Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images , 2020, Precision Agriculture.

[6]  Darius Burschka,et al.  Advances in Computational Stereo , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[7]  Heiko Hirschmüller,et al.  Stereo Processing by Semiglobal Matching and Mutual Information , 2008, IEEE Trans. Pattern Anal. Mach. Intell..

[8]  Jan Dirk Wegner,et al.  Contour Detection in Unstructured 3D Point Clouds , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Asher Bender,et al.  Segmentation of lettuce in coloured 3D point clouds for fresh weight estimation , 2018, Comput. Electron. Agric..

[10]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[11]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[12]  S. Tsaftaris,et al.  Phenotiki: an open software and hardware platform for affordable and easy image‐based phenotyping of rosette‐shaped plants , 2017, The Plant Journal.

[13]  Guilherme N. DeSouza,et al.  Vinobot and Vinoculer: Two Robotic Platforms for High-Throughput Field Phenotyping , 2017, Sensors.

[14]  Achim Walter,et al.  Assessment of Multi-Image Unmanned Aerial Vehicle Based High-Throughput Field Phenotyping of Canopy Temperature , 2020, Frontiers in Plant Science.

[15]  Zhanguo Xin,et al.  Applying genotyping (TILLING) and phenotyping analyses to elucidate gene function in a chemically induced sorghum mutant population , 2008, BMC Plant Biology.

[16]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[17]  Paul Schonfeld,et al.  Heuristic Strategies of Modified Levenberg–Marquardt Algorithm for Fitting Transition Curves , 2020 .

[18]  Patrick S Schnable,et al.  A high-throughput , field-based phenotyping technology for tall biomass crops , 2018 .

[19]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[20]  Stephen Nuske,et al.  StalkNet: A Deep Learning Pipeline for High-Throughput Measurement of Plant Stalk Count and Stalk Width , 2017, FSR.

[21]  Xin Yu,et al.  3D cost aggregation with multiple minimum spanning trees for stereo matching. , 2017, Applied optics.

[22]  Qin Zhang,et al.  A Review of Imaging Techniques for Plant Phenotyping , 2014, Sensors.

[23]  Jin Liu,et al.  Stem–Leaf Segmentation and Phenotypic Trait Extraction of Individual Maize Using Terrestrial LiDAR Data , 2019, IEEE Transactions on Geoscience and Remote Sensing.

[24]  Alex Zelinsky,et al.  Learning OpenCV---Computer Vision with the OpenCV Library (Bradski, G.R. et al.; 2008)[On the Shelf] , 2009, IEEE Robotics & Automation Magazine.

[25]  Adam L. Kaczmarek,et al.  Stereo vision with Equal Baseline Multiple Camera Set (EBMCS) for obtaining depth maps of plants , 2017, Comput. Electron. Agric..

[26]  George Kantor,et al.  The Robotanist: A ground-based agricultural robot for high-throughput crop phenotyping , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[27]  William L. Rooney,et al.  Automated detection and measurement of individual sorghum panicles using density-based clustering of terrestrial lidar data , 2019, ISPRS Journal of Photogrammetry and Remote Sensing.

[28]  Brian L. Steward,et al.  Automated crop plant detection based on the fusion of color and depth images for robotic weed control , 2019, J. Field Robotics.

[29]  David Wettergreen,et al.  In-field segmentation and identification of plant structures using 3D imaging , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[30]  Giulio Reina,et al.  In-field high throughput grapevine phenotyping with a consumer-grade depth camera , 2021, Comput. Electron. Agric..

[31]  Y. Ge,et al.  Principal variable selection to explain grain yield variation in winter wheat from features extracted from UAV imagery , 2019, Plant Methods.

[32]  V. Leemans,et al.  Imaging Wheat Canopy Through Stereo Vision: Overcoming the Challenges of the Laboratory to Field Transition for Morphological Features Extraction , 2020, Frontiers in Plant Science.

[33]  Patrick S. Schnable,et al.  Field‐based robotic phenotyping of sorghum plant architecture using stereo vision , 2018, J. Field Robotics.

[34]  Jayme Garcia Arnal Barbedo,et al.  A Review on the Use of Unmanned Aerial Vehicles and Imaging Sensors for Monitoring and Assessing Plant Stresses , 2019, Drones.

[35]  J. Colorado,et al.  High-Throughput Biomass Estimation in Rice Crops Using UAV Multispectral Imagery , 2019, Journal of Intelligent & Robotic Systems.

[36]  M. Tester,et al.  Phenomics--technologies to relieve the phenotyping bottleneck. , 2011, Trends in plant science.

[37]  ChaivivatrakulSupawadee,et al.  Automatic morphological trait characterization for corn plants via 3D holographic reconstruction , 2014 .

[38]  T. Cheng,et al.  Improved estimation of aboveground biomass in wheat from RGB imagery and point cloud data acquired with a low-cost unmanned aerial vehicle system , 2019, Plant Methods.

[39]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Josep Ramon Morros,et al.  Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry , 2020, Comput. Electron. Agric..

[41]  Heiko Hirschmüller,et al.  Evaluation of Cost Functions for Stereo Matching , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Avideh Zakhor,et al.  Top Down Approach to Height Histogram Estimation of Biomass Sorghum in the Field , 2018, Computational Imaging.

[43]  Changying Li,et al.  Three-dimensional photogrammetric mapping of cotton bolls in situ based on point cloud segmentation and clustering , 2020 .

[44]  Matthew N. Dailey,et al.  Automatic morphological trait characterization for corn plants via 3D holographic reconstruction , 2014 .

[45]  Yongjian Wang,et al.  MVS-Pheno: A Portable and Low-Cost Phenotyping Platform for Maize Shoots Using Multiview Stereo 3D Reconstruction , 2020, Plant phenomics.

[46]  Lav R. Khot,et al.  Evaluation of mobile 3D light detection and ranging based canopy mapping system for tree fruit crops , 2019, Comput. Electron. Agric..

[47]  Jing Zhao,et al.  Genome‐Wide Association Study for Nine Plant Architecture Traits in Sorghum , 2016, The plant genome.