Combining Fractional Order Derivative and Spectral Variable Selection for Organic Matter Estimation of Homogeneous Soil Samples by VIS-NIR Spectroscopy

Visible and near-infrared (VIS–NIR) spectroscopy has been extensively applied to estimate soil organic matter (SOM) in the laboratory. However, if field/moist VIS–NIR spectra can be directly applied to estimate SOM, then much of the time and labor would be avoided. Spectral derivative plays an important role in eliminating unwanted interference and optimizing the estimation model. Nonetheless, the conventional integer order derivatives (i.e., the first and second derivatives) may neglect some detailed information related to SOM. Besides, the full-spectrum generally contains redundant spectral variables, which would affect the model accuracy. This study aimed to investigate different combinations of fractional order derivative (FOD) and spectral variable selection techniques (i.e., competitive adaptive reweighted sampling (CARS), elastic net (ENET) and genetic algorithm (GA)) to optimize the VIS–NIR spectral model of moist soil. Ninety-one soil samples were collected from Central China, with their SOM contents and reflectance spectra measured. Support vector machine (SVM) was applied to estimate SOM. Results indicated that moist spectra differed greatly from dried ground spectra. With increasing order of derivative, the spectral resolution improved gradually, but the spectral strength decreased simultaneously. FOD could provide a better tool to counterbalance the contradiction between spectral resolution and spectral strength. In full-spectrum SVM models, the most accurate estimation was achieved by SVM model based on 1.5-order derivative spectra, with validation R2 = 0.79 and ratio of the performance to deviation (RPD) = 2.20. Of all models studied (different combinations of FOD and variable selection techniques), the highest validation model accuracy for SOM was achieved when applying 1.5 derivative spectra and GA method (validation R2 = 0.88 and RPD = 2.89). Among the three variable selection techniques, overall, the GA method yielded the optimal predictability. However, due to its long computation time, one alternative was to use CARS method. The results of this study confirm that a suitable combination of FOD and variable selection can effectively improve the model performance of SOM in moist soil.

[1]  Jun Wang,et al.  Feasibility of using visible and near-infrared reflectance spectroscopy to monitor heavy metal contaminants in urban lake sediment , 2018 .

[2]  Vladimir Vapnik,et al.  An overview of statistical learning theory , 1999, IEEE Trans. Neural Networks.

[3]  Marston Héracles Domingues Franceschini,et al.  Effects of external factors on soil reflectance measured on-the-go and assessment of potential spectral correction through orthogonalisation and standardisation procedures , 2018 .

[4]  J. Schmitt.  Fractional Derivative Analysis of Diffuse Reflectance Spectra , 1998 .

[5]  C. Jun,et al.  Performance of some variable selection methods when multicollinearity is present , 2005 .

[6]  Riccardo Leardi,et al.  Genetic algorithms in chemometrics , 2012 .

[7]  Michael Vohland,et al.  Determination of soil properties with visible to near- and mid-infrared spectroscopy: Effects of spectral variable selection , 2014 .

[8]  H. Zou,et al.  Regularization and variable selection via the elastic net , 2005 .

[9]  José Alexandre Melo Demattê,et al.  Chemometric soil analysis on the determination of specific bands for the detection of magnesium and potassium by spectroscopy , 2017 .

[10]  Delfim F. M. Torres,et al.  A fractional calculus on arbitrary time scales: Fractional differentiation and fractional integration , 2014, Signal Process..

[11]  Sabine Grunwald,et al.  A systematic study on the application of scatter-corrective and spectral-derivative preprocessing for multivariate prediction of soil organic carbon by Vis-NIR spectra , 2018 .

[12]  Christoph Emmerling,et al.  Determination of total soil organic C and hot water‐extractable C from VIS‐NIR soil reflectance with partial least squares regression and spectral feature selection techniques , 2011 .

[13]  Tao Wang,et al.  Quantitative Model Based on Field-Derived Spectral Characteristics to Estimate Soil Salinity in Minqin County, China , 2014 .

[14]  R. Boggia,et al.  Genetic algorithms as a strategy for feature selection , 1992 .

[15]  R. V. Viscarra Rossel,et al.  rs‐local data‐mines information from spectral libraries to improve local calibrations , 2017 .

[16]  Yufeng Ge,et al.  Penetrometer-mounted VisNIR spectroscopy: Application of EPO-PLS to in situ VisNIR spectra , 2017 .

[17]  B. Minasny,et al.  Removing the effect of soil moisture from NIR diffuse reflectance spectra for the prediction of soil organic carbon , 2011 .

[18]  Roberto Kawakami Harrop Galvão,et al.  A variable elimination method to improve the parsimony of MLR models using the successive projections algorithm , 2008 .

[19]  R. Clark,et al.  Reflectance spectroscopy: Quantitative analysis techniques for remote sensing applications , 1984 .

[20]  Royston Goodacre,et al.  Genetic algorithm optimization for pre-processing and variable selection of spectroscopic data , 2005, Bioinform..

[21]  Zhou Shi,et al.  In situ measurements of organic carbon in soil profiles using vis-NIR spectroscopy on the Qinghai-Tibet plateau. , 2015, Environmental science & technology.

[22]  Manfred F. Buchroithner,et al.  Combining Partial Least Squares and the Gradient-Boosting Method for Soil Property Retrieval Using Visible Near-Infrared Shortwave Infrared Spectra , 2017, Remote. Sens..

[23]  Manjeet Singh,et al.  Development of hyperspectral model for rapid monitoring of soil organic carbon under precision farming in the Indo-Gangetic Plains of Punjab, India , 2015, Journal of the Indian Society of Remote Sensing.

[24]  R. Lal.  Soil carbon sequestration to mitigate climate change , 2004 .

[25]  Bin Li,et al.  Soil mapping via diffuse reflectance spectroscopy based on variable indicators: An ordered predictor selection approach , 2018 .

[26]  Hongdong Li,et al.  Key wavelengths screening using competitive adaptive reweighted sampling method for multivariate calibration. , 2009, Analytica chimica acta.

[27]  Tarin Paz-Kagan,et al.  Linear Multi-Task Learning for Predicting Soil Properties Using Field Spectroscopy , 2017, Remote. Sens..

[28]  R. V. Rossel,et al.  Using data mining to model and interpret soil diffuse reflectance spectra. , 2010 .

[29]  Susana Fernández,et al.  Prediction of Topsoil Organic Carbon Using Airborne and Satellite Hyperspectral Imagery , 2017, Remote. Sens..

[30]  Raphael A. Viscarra Rossel,et al.  Proximal spectral sensing in pedological assessments: vis–NIR spectra for soil classification based on weathering and pedogenesis , 2018 .

[31]  Dong Zhang,et al.  Desert soil clay content estimation using reflectance spectroscopy preprocessed by fractional derivative , 2017, PloS one.

[32]  Eyal Ben-Dor,et al.  Internal soil standard method for the Brazilian soil spectral library: Performance and proximate analysis , 2018 .

[33]  Christina Bogner,et al.  In‐situ prediction of soil organic carbon by vis–NIR spectroscopy: an efficient use of limited field data , 2017 .

[34]  Tiezhu Shi,et al.  Prediction of low heavy metal concentrations in agricultural soils using visible and near-infrared reflectance spectroscopy , 2014 .

[35]  Christoph Emmerling,et al.  Quantification of Soil Variables in a Heterogeneous Soil Region With VIS–NIR–SWIR Data Using Different Statistical Sampling and Modeling Strategies , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[36]  K Wiggins,et al.  An investigation into the use of calculating the first derivative of absorbance spectra as a tool for forensic fibre analysis. , 2007, Science & justice : journal of the Forensic Science Society.

[37]  Yufeng Ge,et al.  Moisture insensitive prediction of soil properties from VNIR reflectance spectra based on external parameter orthogonalization , 2016 .

[38]  Lei Yu,et al.  Prediction of Soil Organic Matter by VIS-NIR Spectroscopy Using Normalized Soil Moisture Index as a Proxy of Soil Moisture , 2017, Remote. Sens..

[39]  Jeffrey A. Andrews,et al.  Soil respiration and the global carbon cycle , 2000 .

[40]  Frans van den Berg,et al.  Review of the most common pre-processing techniques for near-infrared spectra , 2009 .

[41]  Guofeng Wu,et al.  Soil Organic Carbon Content Estimation with Laboratory-Based Visible–Near-Infrared Reflectance Spectroscopy: Feature Selection , 2014, Applied spectroscopy.

[42]  H. Ramon,et al.  On-line measurement of some selected soil properties using a VIS–NIR sensor , 2007 .

[43]  Dong Zhang,et al.  Quantitative Estimating Salt Content of Saline Soil Using Laboratory Hyperspectral Data Treated by Fractional Derivative , 2016 .

[44]  Henning Buddenbaum,et al.  Estimating the soil clay content and organic matter by means of different calibration methods of vis-NIR diffuse reflectance spectroscopy , 2016 .

[45]  Junjie Wang,et al.  Transferability of a Visible and Near-Infrared Model for Soil Organic Matter Estimation in Riparian Landscapes , 2014, Remote. Sens..

[46]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[47]  Zhou Shi,et al.  Accounting for the effects of water and the environment on proximally sensed vis–NIR soil spectra and their calibrations , 2015 .

[48]  Manfred F. Buchroithner,et al.  Quantitative Retrieval of Organic Soil Properties from Visible Near-Infrared Shortwave Infrared (Vis-NIR-SWIR) Spectroscopy Using Fractal-Based Feature Extraction , 2016, Remote. Sens..

[49]  L. Buydens,et al.  Comparing support vector machines to PLS for spectral regression applications , 2004 .

[50]  A. Müller,et al.  Predictions of soil surface and topsoil organic carbon content through the use of laboratory and field spectroscopy in the Albany Thicket Biome of Eastern Cape Province of South Africa , 2011 .

[51]  Meiyan Wang,et al.  Determination of rice root density from Vis–NIR spectroscopy by support vector machine regression and spectral variable selection techniques , 2017 .

[52]  Michael Vohland,et al.  Quantification of Soil Properties with Hyperspectral Data: Selecting Spectral Variables with Different Methods to Improve Accuracies and Analyze Prediction Mechanisms , 2017, Remote. Sens..

[53]  A. Savitzky,et al.  Smoothing and Differentiation of Data by Simplified Least Squares Procedures. , 1964 .

[54]  Sabine Chabrillat,et al.  Evaluating the detection limit of organic matter using point and imaging spectroscopy , 2018, Geoderma.

[55]  Qi Li,et al.  An Efficient Elastic Net with Regression Coefficients Method for Variable Selection of Spectrum Data , 2017, PloS one.

[56]  D. Lobell,et al.  Moisture effects on soil reflectance , 2002 .

[57]  Giorgio Matteucci,et al.  Effect of calibration set size on prediction at local scale of soil carbon by Vis-NIR spectroscopy , 2017 .

[58]  Ting Wu,et al.  Improvement of NIR model by fractional order Savitzky–Golay derivation (FOSGD) coupled with wavelength selection , 2015 .

[59]  Yaqing Ding,et al.  Haar Wavelet Based Implementation Method of the Non–integer Order Differentiation and its Application to Signal Enhancement , 2015 .

[60]  R. Leardi,et al.  Genetic algorithms applied to feature selection in PLS regression: how and when to use them , 1998 .

[61]  Pierre Roudier,et al.  Evaluation of two methods to eliminate the effect of water from soil vis–NIR spectra for predictions of organic carbon , 2017 .

[62]  Christoph Emmerling,et al.  Using Variable Selection and Wavelets to Exploit the Full Potential of Visible–Near Infrared Spectra for Predicting Soil Properties , 2016 .

[63]  Meiyan Wang,et al.  Comparison of multivariate methods for estimating selected soil properties from intact soil cores of paddy fields by Vis–NIR spectroscopy , 2018 .