Effect of regularization term upon fault tolerant training

To enhance fault tolerance of multi-layer neural networks, we proposed PAWMA (partially adaptive weight minimization approach). This method minimizes not only output error but also the sum of squares of weights (the regularization term). This method aims to decrease the number of connections whose faults strongly degrade the performance of MLNs (important connections). On the other hand, weight decay, which aims to eliminate unimportant connections, is base on the same idea. This method expects to keeping important connections and decaying unimportant connections. In this paper, we discuss about the contradiction between two effects of the regularization term. Through some experiment, we show that the difference between two effects is brought by the partially application of the regularization term.

[1]  Geoffrey E. Hinton,et al.  Experiments on Learning by Back Propagation. , 1986 .

[2]  Robert I. Damper,et al.  Determining and improving the fault tolerance of multilayer perceptrons in a pattern-recognition application , 1993, IEEE Trans. Neural Networks.

[3]  T. Haruhiko,et al.  Partially weight minimization approach for fault tolerant multilayer neural networks , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[4]  Haruhiko Takase,et al.  Weight minimization approach for fault tolerant multi-layer neural networks , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[5]  Takehiro Ito,et al.  On fault injection approaches for fault tolerance of feedforward neural networks , 1997, Proceedings Sixth Asian Test Symposium (ATS'97).

[6]  Alan F. Murray,et al.  Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training , 1994, IEEE Trans. Neural Networks.

[7]  G. Bolt,et al.  Fault models for artificial neural networks , 1991, [Proceedings] 1991 IEEE International Joint Conference on Neural Networks.