A Walsh Analysis of Multilayer Perceptron Function

The multilayer perceptron (MLP) is a widely used neural network architecture, but it suffers from the fact that its knowledge representation is not readily interpreted. Hidden neurons take the role of feature detectors, but the popular learning algorithms (back propagation of error, for example) coupled with random starting weights mean that the function implemented by a trained MLP can be difficult to analyse. This paper proposes a method for understanding the structure of the function learned by MLPs that model functions of the class f : f􀀀1;1gn ! Rm. The approach characterises a given MLP using Walsh functions, which make the interactions among subsets of variables explicit. Demonstrations of this analysis used to monitor complexity during learning, understand function structure and measure the generalisation ability of trained networks are presented.

[1]  Donald C. Wunsch,et al.  Neural network explanation using inversion , 2007, Neural Networks.

[2]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990, Proc. IEEE.

[3]  Eric B. Bartlett,et al.  Dynamic node architecture learning: An information theoretic approach , 1994, Neural Networks.

[4]  Juan Julián Merelo Guervós,et al.  Evolving Multilayer Perceptrons , 2000, Neural Processing Letters.

[5]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[6]  Terrence J. Sejnowski,et al.  Analysis of hidden units in a layered network trained to classify sonar targets , 1988, Neural Networks.

[7]  B. Sankur,et al.  Applications of Walsh and related functions , 1986 .

[8]  Wang Jian-guo,et al.  Rule extraction from artificial neural network with optimized activation functions , 2008, 2008 3rd International Conference on Intelligent System and Knowledge Engineering.

[9]  Jeffrey L. Elman,et al.  Finding Structure in Time , 1990, Cogn. Sci..

[10]  Xin Yao,et al.  Evolving artificial neural networks , 1999, Proc. IEEE.

[11]  Belle R. Upadhyaya,et al.  Application of Neural Networks for Sensor Validation and Plant Monitoring , 1990 .

[12]  Nelson F. F. Ebecken,et al.  Extracting rules from multilayer perceptrons in classification problems: A clustering-based approach , 2006, Neurocomputing.

[13]  James L. McClelland,et al.  Parallel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations , 1986 .

[14]  T. Kathirvalavakumar,et al.  Rule extraction from neural networks — A comparative study , 2012, International Conference on Pattern Recognition, Informatics and Medical Engineering (PRIME-2012).

[15]  Lale Özbakir,et al.  Fuzzy DIFACONN-miner: A novel approach for fuzzy rule extraction from neural networks , 2013, Expert Syst. Appl..

[16]  Anders Krogh,et al.  Neural Network Ensembles, Cross Validation, and Active Learning , 1994, NIPS.

[17]  J. Walsh.  A Closed Set of Normal Orthogonal Functions , 1923 .

[18]  Dennis Sanger,et al.  Contribution analysis: a technique for assigning responsibilities to hidden units in connectionist networks , 1991 .

[19]  D. Rumelhart,et al.  Predicting sunspots and exchange rates with connectionist networks , 1991 .

[20]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.