An Adaptive Bayesian Pruning for Neural Networks in a Non-Stationary Environment

Pruning a neural network to a reasonable smaller size, and if possible to give a better generalization, has long been investigated. Conventionally the common technique of pruning is based on considering error sensitivity measure, and the nature of the problem being solved is usually stationary. In this article, we present an adaptive pruning algorithm for use in a nonstationary environment. The idea relies on the use of the extended Kalman filter (EKF) training method. Since EKF is a recursive Bayesian algorithm, we define a weight-importance measure in term of the sensitivity of a posteriori probability. Making use of this new measure and the adaptive nature of EKF, we devise an adaptive pruning algorithm called adaptive Bayesian pruning. Simulation results indicate that in a noisy nonstationary environment, the proposed pruning algorithm is able to remove network redundancy adaptively and yet preserve the same generalization ability.

[1]  Lutz Prechelt,et al.  Connection pruning with static and adaptive pruning schedules , 1997, Neurocomputing.

[2]  Sharad Singhal,et al.  Training Multilayer Perceptrons with the Extende Kalman Algorithm , 1988, NIPS.

[3]  Lee A. Feldkamp,et al.  Neurocontrol of nonlinear dynamical systems with Kalman filter trained recurrent networks , 1994, IEEE Trans. Neural Networks.

[4]  John Moody,et al.  Prediction Risk and Architecture Selection for Neural Networks , 1994 .

[5]  Stefanos Kollias,et al.  An adaptive least squares algorithm for the efficient training of artificial neural networks , 1989 .

[6]  Francesco Palmieri,et al.  Optimal filtering algorithms for fast learning in feedforward neural networks , 1992, Neural Networks.

[7]  D. Mackay,et al.  A Practical Bayesian Framework for Backprop Networks , 1991 .

[8]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[9]  Yann LeCun,et al.  Optimal Brain Damage , 1989, NIPS.

[10]  Lai-Wan Chan,et al.  On-line training and pruning for recursive least square algorithms , 1996 .

[11]  Hideaki Sakai,et al.  A real-time learning algorithm for a multilayered neural network based on the extended Kalman filter , 1992, IEEE Trans. Signal Process..

[12]  Ferdinand Hergert,et al.  Improving model selection by nonconvergent methods , 1993, Neural Networks.

[13]  Eric A. Wan,et al.  Dual Kalman Filtering Methods for Nonlinear Prediction, Smoothing and Estimation , 1996, NIPS.

[14]  Jan Larsen,et al.  DESIGN OF NEURAL NETWORK FILTERS , 1996 .

[15]  B. Anderson,et al.  Optimal Filtering , 1979, IEEE Transactions on Systems, Man, and Cybernetics.

[16]  Lutz Prechelt,et al.  Comparing adaptive and non-adaptive connection pruning with pure early stopping , 1996 .

[17]  Russell Reed,et al.  Pruning algorithms-a survey , 1993, IEEE Trans. Neural Networks.

[18]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.