Features Selection and Architecture Optimization in Connectionist Systems

In this paper, we propose a features selection measure and an architecture optimization procedure for Multi-Layer Perceptrons (MLP). The algorithm presented in this contribution employs a heuristic measure named HVS (Heuristic for Variable Selection). This new measure allows us to identify and select important variables in the features space. This can be achieved by eliminating redundant features and those which do not contain enough relevant information. The proposed measure is used in a new procedure aimed at selecting the "best" MLP architecture given an initial structure. Application results for two generic problems: regression and discrimination, demonstrates the proposed selection algorithm's effectiveness in identifying optimized connectionist models with higher accuracy. Finally, an extension of HVS, named epsilonHVS, is proposed for discriminative features detection and architecture optimization for Time Delay Neural Networks models (TDNN).

[1]  David E. Rumelhart,et al.  Predicting the Future: a Connectionist Approach , 1990, Int. J. Neural Syst..

[2]  Wright-Patterson Afb,et al.  Feature Selection Using a Multilayer Perceptron , 1990 .

[3]  Roberto Battiti,et al.  Using mutual information for selecting features in supervised neural net learning , 1994, IEEE Trans. Neural Networks.

[4]  A. Munoz,et al.  Variable Selection through Statistical Sensibility Analysis: Application to Feedforward and Recurrent Neural Networks. , 1995 .

[5]  Marie Cottrell,et al.  Neural modeling for time series: A statistical stepwise method for weight elimination , 1995, IEEE Trans. Neural Networks.

[6]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[7]  Sylvie Thiria,et al.  NEURAL NETS AND TASK DECOMPOSITION , 1991 .

[8]  Geoffrey E. Hinton,et al.  A time-delay neural network architecture for isolated word recognition , 1990, Neural Networks.

[9]  Leslie G. Valiant,et al.  A theory of the learnable , 1984, STOC '84.

[10]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[11]  Younès Bennani,et al.  HVS : A Heuristic for Variable Selection in Multilayer Artificial Neural Network Classifier , 1997 .

[12]  David J. C. MacKay,et al.  Bayesian Methods for Backpropagation Networks , 1996 .

[13]  Fabrice Rossi.  Attribute Suppression with Multi-Layer Perceptron , 1996 .

[14]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[15]  Ah Chung Tsoi,et al.  Face recognition: a convolutional neural-network approach , 1997, IEEE Trans. Neural Networks.

[16]  Halbert White,et al.  Connectionist nonparametric regression: Multilayer feedforward networks can learn arbitrary mappings , 1990, Neural Networks.

[17]  C. Goutte,et al.  On the use of a pruning prior for neural networks , 1996, Neural Networks for Signal Processing VI. Proceedings of the 1996 IEEE Signal Processing Society Workshop.

[18]  Younès Bennani,et al.  Discriminative feature extraction and selection applied to face recognition , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[19]  Patrick Gallinari,et al.  Variable Selection with Optimal Cell Damage , 1994 .

[20]  Vicki Bruce,et al.  Face Recognition: From Theory to Applications , 1999 .

[21]  L. K. Hansen,et al.  Pruning with generalization based weight saliencies: λOBD, λOBS , 1995, NIPS 1995.

[22]  Lars Kai Hansen,et al.  On design and evaluation of tapped-delay neural network architectures , 1993, IEEE International Conference on Neural Networks.

[23]  Younès Bennani,et al.  Architecture Optimization in Feedforward Connectionist Models , 1998 .