Multilayer Perceptron Learning Utilizing Reducibility Mapping

In the search space of MLP(J), multilayer perceptron having J hidden units, there exist flat areas called singular regions created by applying reducibility mapping to the optimal solution of MLP(J −1). Since such singular regions cause serious slowdown for learning, a learning method for avoiding singular regions has been aspired. However, such avoiding does not guarantee the quality of the final solutions. This paper proposes a new learning method which does not avoid but makes good use of singular regions to stably and successively find solutions excellent enough for MLP(J). The potential of the method is shown by our experiments using artificial and real data sets.

[1]  Ryohei Nakano,et al.  Learning Method Utilizing Singular Region of Multilayer Perceptron , 2011, IJCCI.

[2]  Kazumi Saito,et al.  Partial BFGS Update and Efficient Step-Length Calculation for Three-Layer Neural Networks , 1997, Neural Computation.

[3]  Kenji Fukumizu,et al.  Local minima and plateaus in hierarchical structures of multilayer perceptrons , 2000, Neural Networks.

[4]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[5]  Kazumi Saito,et al.  Discovering Polynomials to Fit Multivariate Data Having Numeric and Nominal Variables , 2002, Progress in Discovery Science.

[6]  Héctor J. Sussmann,et al.  Uniqueness of the weights for minimal feedforward nets with a given input-output map , 1992, Neural Networks.

[7]  Sumio Watanabe.  Algebraic Geometry and Statistical Learning Theory , 2009 .

[8]  David G. Stork,et al.  Pattern Classification (2nd ed.) , 1999 .

[9]  Kenji Fukumizu,et al.  Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons , 2000, Neural Computation.

[10]  Robert Hecht-Nielsen,et al.  Neural network tomography: Network replication from output surface geometry , 2011, Neural Networks.

[11]  Leonard G. C. Hamey,et al.  XOR has no local minima: A case study in neural network error surface analysis , 1998, Neural Networks.

[12]  Setsuo Arikawa,et al.  Progress in Discovery Science , 2002, Lecture Notes in Computer Science.

[13]  Chen-Han Sung.  Temporal knowledge: Recognition and learning of time-based patterns , 1988, Neural Networks.

[14]  David G. Luenberger,et al.  Linear and nonlinear programming , 1984 .