Fractal measures and diffusion as results of learning in neural networks

Abstract It is shown that learning in neural networks can result in multifractal invariant distributions in parameter space if the patterns of the learning set are presented at random. Lowering the learning rate entails a transition to distributions with Euclidian support. For backpropagation learning a comparison with a Fokker-Planck approximation shows the practical relevance of the parameter-dependent diffusion tensor.

[1]  Sompolinsky,et al.  Learning from examples in large neural networks. , 1990, Physical review letters.

[2]  Pal Rujan,et al.  Strange attractors in disordered systems , 1984 .

[3]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[4]  Kenneth Falconer,et al.  The Hausdorff dimension of some fractals and attractors of overlapping construction , 1987 .

[5]  H. Ritter,et al.  Convergence properties of Kohonen's topology conserving maps: fluctuations, stability, and dimension selection , 1988, Biological Cybernetics.

[6]  David J. Burr,et al.  Experiments on neural net recognition of spoken and written text , 1988, IEEE Trans. Acoust. Speech Signal Process..

[7]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[8]  Michael F. Barnsley,et al.  Fractals everywhere , 1988 .

[9]  Grebogi,et al.  Multifractal properties of snapshot attractors of random maps. , 1990, Physical review. A, Atomic, molecular, and optical physics.

[10]  B. Mandelbrot.  Fractal Geometry of Nature , 1984 .

[11]  F. Martinelli,et al.  Random composition of two rational maps: Singularity of the invariant measure , 1988 .

[12]  Fractal measures in the random-field Ising model , 1987 .

[13]  S. Karlin.  Some random walks arising in learning models. I. , 1953 .

[14]  Ulrich Behn,et al.  One-dimensional random field Ising model and discrete stochastic mappings , 1987 .

[15]  Alexander H. Waibel,et al.  Modular Construction of Time-Delay Neural Networks for Speech Recognition , 1989, Neural Computation.

[16]  P. Erdös.  On a Family of Symmetric Bernoulli Convolutions , 1939 .

[17]  Halbert White,et al.  Learning in Artificial Neural Networks: A Statistical Perspective , 1989, Neural Computation.

[18]  Roberto Benzi,et al.  On the multifractal nature of fully developed turbulence and chaotic systems , 1984 .

[19]  Jensen,et al.  Erratum: Fractal measures and their singularities: The characterization of strange sets , 1986, Physical review. A, General physics.

[20]  Robijn Bruinsma,et al.  One-dimensional Ising model in a random field , 1983 .

[21]  P. Hänggi,et al.  Discrete dynamics and metastability: Mean first passage times and escape rates , 1987 .

[22]  H. G. E. Hentschel,et al.  The infinite number of generalized dimensions of fractals and strange attractors , 1983 .

[23]  H. Kushner.  Asymptotic global behavior for stochastic approximation and diffusions with slowly decreasing noise effects: Global minimization via Monte Carlo , 1987 .