Design and Application of a Variable Selection Method for Multilayer Perceptron Neural Network With LASSO

In this paper, a novel variable selection method for neural network that can be applied to describe nonlinear industrial processes is developed. The proposed method is an iterative two-step approach. First, a multilayer perceptron is constructed. Second, the least absolute shrinkage and selection operator is introduced to select the input variables that are truly essential to the model with the shrinkage parameter is determined using a cross-validation method. Then, variables whose input weights are zero are eliminated from the data set. The algorithm is repeated until there is no improvement in the model accuracy. Simulation examples as well as an industrial application in a crude distillation unit are used to validate the proposed algorithm. The results show that the proposed approach can be used to construct a more compressed model, which incorporates a higher level of prediction accuracy than other existing methods.

[1]  G. P. Liu,et al.  Nonlinear Identification and Control: A Neural Network Approach [Book Review] , 2002, IEEE Control Systems.

[2]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[3]  Nenad Bolf,et al.  Soft Sensors for Kerosene Properties Estimation and Control in Crude Distillation Unit , 2009 .

[4]  Rasmus Bro,et al.  Variable selection in regression—a tutorial , 2010 .

[5]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[6]  Jie Zhang,et al.  Inferential estimation of polymer quality using bootstrap aggregated neural networks , 1999, Neural Networks.

[7]  Christian Lebiere,et al.  The Cascade-Correlation Learning Architecture , 1989, NIPS.

[8]  Weizhong Yan,et al.  Toward Automatic Time-Series Forecasting Using Neural Networks , 2012, IEEE Transactions on Neural Networks and Learning Systems.

[9]  Yongdai Kim,et al.  Smoothly Clipped Absolute Deviation on High Dimensions , 2008 .

[10]  Francis J. Doyle,et al.  Neural network-based software sensor: training set design and application to a continuous pulp digester , 2005 .

[11]  Hiromasa Kaneko,et al.  Nonlinear regression method with variable region selection and application to soft sensors , 2013 .

[12]  J. A. Pérez-Benitez,et al.  Feature Selection and Neural Network for analysis of microstructural changes in magnetic materials , 2011, Expert Syst. Appl..

[13]  Luigi Fortuna,et al.  Soft sensors for product quality monitoring in debutanizer distillation columns , 2005 .

[14]  Gareth M. James,et al.  Improved variable selection with Forward-Lasso adaptive shrinkage , 2011, 1104.3390.

[15]  Maria Gabriella Xibilia,et al.  FPGA based soft sensor for the estimation of the kerosene freezing point , 2009, 2009 IEEE International Symposium on Industrial Embedded Systems.

[16]  Jeanny Hérault,et al.  Curvilinear component analysis: a self-organizing neural network for nonlinear mapping of data sets , 1997, IEEE Trans. Neural Networks.

[17]  Jacek M. Zurada,et al.  Normalized Mutual Information Feature Selection , 2009, IEEE Transactions on Neural Networks.

[18]  J. R. Whiteley,et al.  Development of inferential measurements using neural networks. , 2001, ISA transactions.

[19]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[20]  Nenad Bolf,et al.  Distillation End Point Estimation in Diesel Fuel Production , 2013 .

[21]  Eric Fock,et al.  Global Sensitivity Analysis Approach for Input Selection and System Identification Purposes—A New Framework for Feedforward Neural Networks , 2014, IEEE Transactions on Neural Networks and Learning Systems.

[22]  Luiz Augusto da Cruz Meleiro,et al.  ANN-based soft-sensor for real-time process monitoring and control of an industrial polymerization process , 2009, Comput. Chem. Eng..

[23]  Guilherme De A. Barreto,et al.  Long-term time series prediction with the NARX network: An empirical evaluation , 2008, Neurocomputing.

[24]  Nikhil R. Pal,et al.  Feature Selection Using a Neural Framework With Controlled Redundancy , 2015, IEEE Transactions on Neural Networks and Learning Systems.

[25]  Vijander Singh,et al.  Development of soft sensor for neural network based control of distillation column. , 2013, ISA transactions.

[26]  Thomas F. Coleman,et al.  An Interior Trust Region Approach for Nonlinear Minimization Subject to Bounds , 1993, SIAM J. Optim..

[27]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.

[28]  Rui Araújo,et al.  A multilayer-perceptron based method for variable selection in soft sensor design , 2013 .

[29]  Roberto Battiti,et al.  Using mutual information for selecting features in supervised neural net learning , 1994, IEEE Trans. Neural Networks.

[30]  David R. Anderson,et al.  Model selection and multimodel inference : a practical information-theoretic approach , 2003 .

[31]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[32]  James A. Rodger,et al.  A fuzzy nearest neighbor neural network statistical model for predicting demand for natural gas and energy cost savings in public buildings , 2014, Expert Syst. Appl..

[33]  H. Zou.  The Adaptive Lasso and Its Oracle Properties , 2006 .

[34]  Ron Kohavi,et al.  A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection , 1995, IJCAI.

[35]  Josep M. Sopena,et al.  Performing Feature Selection With Multilayer Perceptrons , 2008, IEEE Transactions on Neural Networks.

[36]  Jianqing Fan,et al.  Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties , 2001 .

[37]  Jianqing Fan,et al.  A Selective Overview of Variable Selection in High Dimensional Feature Space. , 2009, Statistica Sinica.