Unsupervised learning with stochastic gradient

A stochastic gradient is formulated based on deterministic gradient augmented with Cauchy simulated annealing capable to reach a global minimum with a convergence speed significantly faster when simulated annealing is used alone. In order to solve space-time variant inverse problems known as blind source separation, a novel Helmholtz free energy contrast function, H=E-T"0S, with imposed thermodynamics constraint at a constant temperature T"0 was introduced generalizing the Shannon maximum entropy S of the closed systems to the open systems having non-zero input-output energy exchange E. Here, only the input data vector was known while source vector and mixing matrix were unknown. A stochastic gradient was successfully applied to solve inverse space-variant imaging problems on a concurrent pixel-by-pixel basis with the unknown mixing matrix (imaging point spread function) varying from pixel to pixel.

[1]  Harold Szu.  Thermodynamics Energy for both Supervised and Unsupervised Learning Neural Nets at a Constant Temperature , 1999, Int. J. Neural Syst..

[2]  Ivica Kopriva,et al.  Space-Time Variant Blind Source Separation with Additive Noise , 2004, ICA.

[3]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[4]  P. Comon.  Independent Component Analysis , 1992 .

[5]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[6]  A. Guyton,et al.  Textbook of Medical Physiology , 1961 .

[7]  E. Mizutani,et al.  Neuro-Fuzzy and Soft Computing-A Computational Approach to Learning and Machine Intelligence [Book Review] , 1997, IEEE Transactions on Automatic Control.

[8]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[9]  Harold H. Szu,et al.  DETERMINISTIC BLIND SOURCE SEPARATION FOR SPACE VARIANT IMAGING , 2003 .

[10]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  J. Cardoso.  Infomax and maximum likelihood for blind source separation , 1997, IEEE Signal Processing Letters.

[12]  D. Chakrabarti,et al.  A fast fixed - point algorithm for independent component analysis , 1997 .

[13]  E. Fava.  Sensation & Perception , 2004, NeuroImage.

[14]  S. Grossberg,et al.  Adaptive pattern classification and universal recoding: I. Parallel development and coding of neural feature detectors , 1976, Biological Cybernetics.

[15]  Richard Durbin,et al.  An analogue approach to the travelling salesman problem using an elastic net method , 1987, Nature.

[16]  Alan F. Murray,et al.  International Joint Conference on Neural Networks , 1993 .

[17]  Gilles Burel,et al.  Blind separation of sources: A nonlinear neural algorithm , 1992, Neural Networks.

[18]  Jean-François Cardoso,et al.  Equivariant adaptive source separation , 1996, IEEE Trans. Signal Process..

[19]  Harold H. Szu,et al.  Blind inversion in nonlinear space-variant imaging using Cauchy machine , 2003, SPIE Defense + Commercial Sensing.

[20]  B. Sakmann,et al.  Single-channel currents recorded from membrane of denervated frog muscle fibres , 1976, Nature.

[21]  N. Metropolis,et al.  Equation of State Calculations by Fast Computing Machines , 1953, Resonance.

[22]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[23]  J. J. Hopfield,et al.  “Neural” computation of decisions in optimization problems , 1985, Biological Cybernetics.

[24]  Elizabeth Antébi,et al.  Biotechnology: Strategies for Life , 1986 .

[25]  GrossbergS..  Adaptive pattern classification and universal recoding , 1976 .

[26]  Andrzej Cichocki,et al.  Adaptive Blind Signal and Image Processing - Learning Algorithms and Applications , 2002 .

[27]  Andrzej Cichocki,et al.  Adaptive blind signal and image processing , 2002 .

[28]  James R. Buss,et al.  Early tumor detection by multiple infrared unsupervised neural nets fusion , 2003, Proceedings of the 25th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (IEEE Cat. No.03CH37439).

[29]  H. Szu,et al.  Nonconvex optimization by fast simulated annealing , 1987, Proceedings of the IEEE.

[30]  Helena Curtis,et al.  Invitation to Biology , 1972 .

[31]  H. Szu,et al.  Comparison of Lagrange constrained neural network with traditional ICA methods , 2002, Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN'02 (Cat. No.02CH37290).

[32]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[33]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[34]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[35]  Shigeo Abe DrEng.  Pattern Classification , 2001, Springer London.

[36]  David G. Stork,et al.  Pattern Classification , 1973 .

[37]  SorouchyariF..  Blind separation of sources, Part III , 1991 .

[38]  Ali Mansour,et al.  Blind Separation of Sources , 1999 .

[39]  Christian Jutten,et al.  Source separation in post-nonlinear mixtures , 1999, IEEE Trans. Signal Process..

[40]  Donald Geman,et al.  Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images , 1984 .

[41]  H Szu,et al.  Landsat spectral demixing a la superresolution of blind matrix inversion by constraint MaxEnt neural nets , 1997, Defense, Security, and Sensing.

[42]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[43]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[44]  Harold H. Szu,et al.  Artificial neural networks for noisy image super-resolution , 2001 .

[45]  Philippe Garat,et al.  Blind separation of mixture of independent sources through a quasi-maximum likelihood approach , 1997, IEEE Trans. Signal Process..

[46]  E. M.,et al.  Statistical Mechanics , 2021, Manual for Theoretical Chemistry.

[47]  Shun-ichi Amari,et al.  Information geometry of neural network—an overview , 1997 .

[48]  J. Cardoso,et al.  Blind beamforming for non-gaussian signals , 1993 .

[49]  Dinh-Tuan Pham,et al.  Improving algorithm speed in PNL mixture separation and Wiener system inversion , 2003 .