Neural network tomography: Network replication from output surface geometry

Multilayer perceptron networks whose outputs consist of affine combinations of hidden units using the tanh activation function are universal function approximators and are used for regression, typically by reducing the MSE with backpropagation. We present a neural network weight learning algorithm that directly positions the hidden units within input space by numerically analyzing the curvature of the output surface. Our results show that under some sampling requirements, this method can reliably recover the parameters of a neural network used to generate a data set.

[1]  G. Farin.  Curves and Surfaces for Cagd: A Practical Guide , 2001 .

[2]  S. Stigler.  Gauss and the Invention of Least Squares , 1981 .

[3]  Héctor J. Sussmann,et al.  Uniqueness of the weights for minimal feedforward nets with a given input-output map , 1992, Neural Networks.

[4]  Paul J. Werbos,et al.  The roots of backpropagation , 1994 .

[5]  Robert Hecht-Nielsen,et al.  On the Geometry of Feedforward Neural Network Error Surfaces , 1993, Neural Computation.

[6]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[7]  Shun-ichi Amari,et al.  A Theory of Adaptive Pattern Classifiers , 1967, IEEE Trans. Electron. Comput..

[8]  Eric A. Wan,et al.  Neural network classification: a Bayesian interpretation , 1990, IEEE Trans. Neural Networks.

[9]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[10]  I. Daubechies.  Orthonormal bases of compactly supported wavelets , 1988 .

[11]  John G. Daugman,et al.  Complete discrete 2-D Gabor transforms by neural networks for image analysis and compression , 1988, IEEE Trans. Acoust. Speech Signal Process..

[12]  Nils J. Nilsson,et al.  Learning Machines: Foundations of Trainable Pattern-Classifying Systems , 1965 .

[13]  A. G. Ivakhnenko,et al.  Polynomial Theory of Complex Systems , 1971, IEEE Trans. Syst. Man Cybern..

[14]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[15]  Alexander I. Galushkin,et al.  Neural Networks Theory , 2007 .

[16]  Bruce W. Suter,et al.  The multilayer perceptron as an approximation to a Bayes optimal discriminant function , 1990, IEEE Trans. Neural Networks.

[17]  Frank Rosenblatt,et al.  PRINCIPLES OF NEURODYNAMICS. PERCEPTRONS AND THE THEORY OF BRAIN MECHANISMS , 1963 .

[18]  Simon Haykin,et al.  Neural Networks and Learning Machines , 2010 .

[19]  Adrien-Marie Legendre,et al.  Nouvelles méthodes pour la détermination des orbites des comètes , 1970 .

[20]  Dennis Gabor,et al.  Theory of communication , 1946 .

[21]  Paul C. Kainen,et al.  Functionally Equivalent Feedforward Neural Networks , 1994, Neural Computation.

[22]  James A. Anderson,et al.  Talking Nets: An Oral History of Neurocomputing , 1998 .

[23]  Sung-Kwun Oh,et al.  The design of self-organizing Polynomial Neural Networks , 2002, Inf. Sci..

[24]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[25]  Karl Steinbuch,et al.  Automat und Mensch , 1962 .

[26]  J. Fourier.  Théorie analytique de la chaleur , 2009 .

[27]  Kenji Fukumizu,et al.  Local minima and plateaus in hierarchical structures of multilayer perceptrons , 2000, Neural Networks.