Backward Elimination Methods for Associative Memory Network Pruning

Three hybrid data based model construction/pruning formula are introduced by using backward elimination as automatic postprocessing approaches to improved model sparsity. Each of these approaches is based on a composite cost function between the model fit and one of three terms of A-/D-optimality / (parameter 1-norm in basis pursuit) that determines a pruning process. The A-/D-optimality based pruning formula contain some orthogonalisation between the pruned model and the deleted regressor. The basis pursuit cost function is derived as a simple formula without need for an orthogonalisation process. These different approaches to parsimonious data based modelling are applied to the same numerical examples in parallel to demonstrate their computational effectiveness.

[1]  S. Billings,et al.  Givens rotation based fast backward elimination algorithm for RBF neural network pruning , 1997 .

[2]  Xia Hong,et al.  A neurofuzzy network knowledge extraction and extended Gram-Schmidt algorithm for model subspace decomposition , 2003, IEEE Trans. Fuzzy Syst..

[3]  Anthony C. Atkinson,et al.  Optimum Experimental Designs , 1992 .

[4]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[5]  Sheng Chen,et al.  Combined genetic algorithm optimization and regularized orthogonal least squares learning for radial basis function networks , 1999, IEEE Trans. Neural Networks.

[6]  Roderick Murray-Smith,et al.  Multiple Model Approaches to Modelling and Control , 1997 .

[7]  Xia Hong,et al.  Nonlinear model structure design and construction using orthogonal least squares and D-optimality design , 2002, IEEE Trans. Neural Networks.

[8]  Xia Hong,et al.  Adaptive Modelling, Estimation and Fusion from Data: A Neurofuzzy Approach , 2002, Advanced information processing.

[9]  Xia Hong,et al.  Nonlinear model structure detection using optimum experimental design and orthogonal least squares , 2001, IEEE Trans. Neural Networks.

[10]  CentresMark,et al.  Regularisation in the Selection of Radial Basis Function , 1995 .

[11]  Michael A. Saunders,et al.  Atomic Decomposition by Basis Pursuit , 1998, SIAM J. Sci. Comput..

[12]  P. Laycock,et al.  Optimum Experimental Designs , 1995 .

[13]  Martin Brown,et al.  Neurofuzzy adaptive modelling and control , 1994 .

[14]  Sheng Chen,et al.  Orthogonal least squares methods and their application to non-linear system identification , 1989 .

[15]  R. Tibshirani,et al.  Least angle regression , 2004, math/0406456.