Evaluating calibration methods for predicting soil available nutrients using hyperspectral VNIR data

Abstract Soil nutrients, including available nitrogen (N), phosphorous (P), and potassium (K), are critical properties for monitoring soil fertility and function. Spectroscopy analysis has proven to be a rapid and effective means for predicting soil properties, in general, and NPK, in particular. However, different calibration methods, including preprocessing transformations (PPTs) and regression algorithms (RAs), considerably affect the performance of prediction models. In this study, raw spectrum and 21 PPTs, combined with three RAs, for a total of 66 calibration methods, were investigated for modeling and predicting soil NPK using hyperspectral VNIR data (400–1000 nm). The ratio of performance to deviation (RPD) of validation set was selected to evaluate the prediction accuracy and the ratio between the interpretable sum squared deviation and the real sum squared deviation (SSR/SST) of the validation set was also used to evaluate the explanatory power of the models. It was found that there is a tradeoff between RPD and SSR/SST values; under this tradeoff, the multiplicative scatter correction, combined with the back-propagation neural network, was preferred for predicting P (RPD = 2.23, SSR/SST = 0.81). The Savitzky-Golay filtering + logarithmic transformation, combined with the partial least squares – regression, was preferred for predicting K (RPD = 1.47, SSR/SST = 0.95). However, with extremely low RPD and SSR/SST values, the prediction of N was unreliable in this study. The evaluation approach presented in this paper suggests a framework for choosing a calibration method for spectroscopy analysis for predicting soil NPK and perhaps some other properties.

[1]  Bridget Fowler,et al.  A Sociological Analysis of the Satanic Verses Affair , 2000 .

[2]  Galit Shmueli,et al.  To Explain or To Predict? , 2010, 1101.0891.

[3]  R. V. Rossel,et al.  Visible, near infrared, mid infrared or combined diffuse reflectance spectroscopy for simultaneous assessment of various soil properties , 2006 .

[4]  H. Martens,et al.  Variable Selection in near Infrared Spectroscopy Based on Significance Testing in Partial Least Squares Regression , 2000 .

[5]  J. Newhouse Is Collinearity a Problem , 1971 .

[6]  Simon R. Arridge,et al.  Data analysis methods for near-infrared spectroscopy of tissue: problems in determining the relative cytochrome aa3 concentration , 1991, Photonics West - Lasers and Applications in Science and Engineering.

[7]  C. Hurburgh,et al.  Near-Infrared Reflectance Spectroscopy–Principal Components Regression Analyses of Soil Properties , 2001 .

[8]  Tormod Næs,et al.  Related versions of the multiplicative scatter correction method for preprocessing spectroscopic data , 1995 .

[9]  L Galvdo,et al.  Relationships of spectral reflectance and color among surface and subsurface horizons of tropical soil profiles , 1997 .

[10]  D. Prezioso,et al.  Laboratory Assessment , 2007, Urologia Internationalis.

[11]  Jingyi Huang,et al.  Correction: In Situ Measurement of Some Soil Properties in Paddy Soil Using Visible and Near-Infrared Spectroscopy , 2016, PloS one.

[12]  J. Duckworth Mathematical Data Preprocessing , 2015 .

[13]  J. M. Soriano-Disla,et al.  The Performance of Visible, Near-, and Mid-Infrared Reflectance Spectroscopy for Prediction of Soil Physical, Chemical, and Biological Properties , 2014 .

[14]  Andrej-Nikolai Spiess,et al.  An evaluation of R2 as an inadequate measure for nonlinear models in pharmacological and biochemical research: a Monte Carlo approach , 2010, BMC pharmacology.

[15]  R. V. Rossel,et al.  Visible and near infrared spectroscopy in soil science , 2010 .

[16]  L. A. Stone,et al.  Computer Aided Design of Experiments , 1969 .

[17]  Mohammadmehdi Saberioon,et al.  Comparing different data preprocessing methods for monitoring soil heavy metals based on soil spectral features. , 2016 .

[18]  S. Savcı An Agricultural Pollutant: Chemical Fertilizer , 2012 .

[19]  Rebecca L. Whetton,et al.  Machine learning based prediction of soil total nitrogen, organic carbon and moisture content by using VIS-NIR spectroscopy , 2016 .

[20]  Yong He,et al.  Nitrogen, phosphorus, and potassium prediction in soils, using infrared spectroscopy , 2011 .

[21]  J. Osborne Prediction in Multiple Regression , 2000 .

[22]  D. Cozzolino,et al.  The prediction of total anthocyanin concentration in red-grape homogenates using visible-near-infrared spectroscopy and artificial neural networks. , 2007, Analytica chimica acta.

[23]  R. Clark,et al.  High spectral resolution reflectance spectroscopy of minerals , 1990 .

[24]  Jiansheng Wu,et al.  Soil moisture retrieving using hyperspectral data with the application of wavelet analysis , 2013, Environmental Earth Sciences.

[25]  S. Wold,et al.  The Collinearity Problem in Linear Regression. The Partial Least Squares (PLS) Approach to Generalized Inverses , 1984 .

[26]  Henning Buddenbaum,et al.  The Effects of Spectral Pretreatments on Chemometric Analyses of Soil Profiles Using Laboratory Imaging Spectroscopy , 2012 .

[27]  Ehsan Mesbahi,et al.  Artificial neural networks: fundamentals , 2003 .

[28]  Eyal Ben-Dor,et al.  Mapping the Spectral Soil Quality Index (SSQI) Using Airborne Imaging Spectroscopy , 2015, Remote. Sens..

[29]  Jerome J. Workman,et al.  Near-infrared spectroscopy in agriculture , 2004 .

[30]  Yoel Shkolnisky,et al.  Change detection of soils under small-scale laboratory conditions using imaging spectroscopy sensors , 2014 .

[31]  Xinjun Peng,et al.  TSVR: An efficient Twin Support Vector Machine for regression , 2010, Neural Networks.

[32]  J. Tukey,et al.  Variations of Box Plots , 1978 .

[33]  Xueguang Shao,et al.  Continuous Wavelet Transform Applied to Removing the Fluctuating Background in Near-Infrared Spectra , 2004, J. Chem. Inf. Model..

[34]  Kaoru Hirota,et al.  Improving recognition and generalization capability of back-propagation NN using a self-organized network inspired by immune algorithm (SONIA) , 2005, Appl. Soft Comput..

[35]  Yan Guo,et al.  [Applying local neural network and visible/near-infrared spectroscopy to estimating available nitrogen, phosphorus and potassium in soil]. , 2014, Guang pu xue yu guang pu fen xi = Guang pu.

[36]  G. Hunt Visible and near-infrared spectra of minerals and rocks : I silicate minerals , 1970 .

[37]  H. Ramon,et al.  Comparison among principal component, partial least squares and back propagation neural network analyses for accuracy of measurement of selected soil properties with visible and near infrared spectroscopy , 2010 .

[38]  Frans van den Berg,et al.  Review of the most common pre-processing techniques for near-infrared spectra , 2009 .

[39]  Guofeng Wu,et al.  Comparison of multivariate methods for estimating soil total nitrogen with visible/near-infrared spectroscopy , 2012, Plant and Soil.

[40]  Zhou Shi,et al.  Sensing soil condition and functions , 2016 .

[41]  C. Hua Study on Wheat Yield Stability in Huaibei Lime Concretion Black Soil Area Based on Long-Term Fertilization Experiment , 2014 .

[42]  Haiyan Cen,et al.  Nondestructive detection of chilling injury in cucumber fruit using hyperspectral imaging with feature selection and supervised classification , 2016 .

[43]  Jun Zhou,et al.  Soil Biochar Quantification via Hyperspectral Unmixing , 2013, 2013 International Conference on Digital Image Computing: Techniques and Applications (DICTA).

[44]  F. Fornasier,et al.  The Potential of near Infrared Reflectance Spectroscopy as a Tool for the Chemical Characterisation of Agricultural Soils , 2001 .

[45]  I A Basheer,et al.  Artificial neural networks: fundamentals, computing, design, and application. , 2000, Journal of microbiological methods.

[46]  M. C. Sarathjith,et al.  Comparison of data mining approaches for estimating soil nutrient contents using diffuse reflectance spectroscopy , 2015 .

[47]  Dazhou Zhu,et al.  [The Classification of Wheat Varieties Based on Near Infrared Hyperspectral Imaging and Information Fusion]. , 2015, Guang pu xue yu guang pu fen xi = Guang pu.

[48]  Xiang Yu,et al.  Evaluation of MLSR and PLSR for estimating soil element contents using visible/near-infrared spectroscopy in apple orchards on the Jiaodong peninsula , 2016 .

[49]  Xi Ji,et al.  Exergetic assessment for ecological economic system: Chinese agriculture , 2009 .

[50]  A. M. Mouazen,et al.  Data fusion techniques for delineation of site-specific management zones in a field in UK , 2015, Precision Agriculture.

[51]  Weixing Cao,et al.  Laboratory assessment of three quantitative methods for estimating the organic matter content of soils in China based on visible/near-infrared reflectance spectra , 2013 .

[52]  Tao Wang,et al.  Quantitative Model Based on Field-Derived Spectral Characteristics to Estimate Soil Salinity in Minqin County, China , 2014 .

[53]  A. Karnieli,et al.  A spectral soil quality index (SSQI) for characterizing soil function in areas of changed land use , 2014 .

[54]  Michael Vohland,et al.  Use of A Portable Camera for Proximal Soil Sensing with Hyperspectral Image Data , 2015, Remote. Sens..

[55]  Andrej-Nikolai Spiess,et al.  articleAn evaluation of R 2 as an inadequate measure for nonlinear models in pharmacological and biochemical research : a Monte Carlo approach , 2015 .

[56]  A. Savitzky,et al.  Smoothing and Differentiation of Data by Simplified Least Squares Procedures. , 1964 .

[57]  Kenneth A. Sudduth,et al.  Soil Phosphorus and Potassium Estimation by Reflectance Spectroscopy , 2016 .

[58]  Roman M. Balabin,et al.  Variable selection in near-infrared spectroscopy: benchmarking of feature selection methods on biodiesel data. , 2011, Analytica chimica acta.

[59]  Jean-Philippe Gras,et al.  Best practices for obtaining and processing field visible and near infrared (VNIR) spectra of topsoils , 2014 .

[60]  Johan A. K. Suykens,et al.  Least Squares Support Vector Machines , 2002 .

[61]  S. Weisberg,et al.  Applied Linear Regression (2nd ed.). , 1986 .

[62]  N. Holden,et al.  Determination of Soil Organic Matter and Carbon Fractions in Forest Top Soils using Spectral Data Acquired from Visible–Near Infrared Hyperspectral Images , 2012 .

[63]  H. Ramon,et al.  Potential for On-Site Analysis of Hog Manure Using a Visual and near Infrared Diode Array Reflectance Spectrometer , 2004 .

[64]  S. Weisberg Applied Linear Regression: Weisberg/Applied Linear Regression 3e , 2005 .

[65]  Tom Fearn,et al.  Practical Nir Spectroscopy With Applications in Food and Beverage Analysis , 1993 .

[66]  A. Gholizadeh,et al.  Visible and near infrared reflectance spectroscopy to determine chemical properties of paddy soils , 2013 .

[67]  Johan A. K. Suykens,et al.  Coupled Simulated Annealing , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[68]  T. Hassard,et al.  Applied Linear Regression , 2005 .

[69]  S. Wold,et al.  The multivariate calibration problem in chemistry solved by the PLS method , 1983 .

[70]  T. O. Kvålseth Cautionary Note about R 2 , 1985 .

[71]  Xiaowei Yang,et al.  A heuristic weight-setting strategy and iteratively updating algorithm for weighted least-squares support vector regression , 2008, Neurocomputing.

[72]  Eyal Ben-Dor,et al.  Near-Infrared Analysis as a Rapid Method to Simultaneously Evaluate Several Soil Properties , 1995 .

[73]  Zongjian Lin,et al.  Spectral features of soil organic matter , 2009 .