Blind signal processing by the adaptive activation function neurons

The aim of this paper is to study an Information Theory based learning theory for neural units endowed with adaptive activation functions. The learning theory has the target to force the neuron to approximate the input-output transference that makes it flat (uniform) the probability density function of its output or, equivalently, that maximizes the entropy of the neuron response. Then, a network of adaptive activation function neurons is studied, and the effectiveness of the new structure is tested on Independent Component Analysis (ICA) problems. The new ICA neural algorithm is compared with the closely related 'Mixture of Densities' (MOD) technique by Xu et al.. Both simulation results and structural comparison show the new method is effective and more efficient in computational complexity.

[1]  Ralph Linsker,et al.  An Application of the Principle of Maximum Information Preservation to Linear Systems , 1988, NIPS.

[2]  Simon Haykin,et al.  Neural networks expand SP's horizons , 1996, IEEE Signal Process. Mag..

[3]  Pierre Comon,et al.  Independent component analysis, A new concept? , 1994, Signal Process..

[4]  Shun-ichi Amari,et al.  Independent component analysis by the information-theoretic approach with mixture of densities , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[5]  Vladimir Vapnik,et al.  The Support Vector Method , 1997, ICANN.

[6]  Terrence J. Sejnowski,et al.  An Information-Maximization Approach to Blind Separation and Blind Deconvolution , 1995, Neural Computation.

[7]  Shun-ichi Amari,et al.  Nonlinearity and separation capability: further justification for the ICA algorithm with mixture of densities , 1997, ESANN.

[8]  D. Signorini,et al.  Neural networks , 1995, The Lancet.

[9]  Yoh-Han Pao,et al.  Adaptive pattern recognition and neural networks , 1989 .

[10]  S. Amari,et al.  Nonlinearity and Separation Capability: Further Justiication for the Ica Algorithm with a Learned Mixture of Parametric Densities , 1997 .

[11]  Ralph Linsker,et al.  Local Synaptic Learning Rules Suffice to Maximize Mutual Information in a Linear Network , 1992, Neural Computation.

[12]  Simone G. O. Fiori,et al.  Non-uniform image sampling for robot motion control by the GFS neural algorithm , 1999, IJCNN'99. International Joint Conference on Neural Networks. Proceedings (Cat. No.99CH36339).

[13]  Christian Jutten,et al.  Blind separation of sources, part I: An adaptive algorithm based on neuromimetic architecture , 1991, Signal Process..

[14]  Erkki Oja,et al.  Applications of neural blind separation to signal and image processing , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[15]  Noga Alon,et al.  Source coding and graph entropies , 1996, IEEE Trans. Inf. Theory.

[16]  J. Nadal,et al.  Nonlinear feedforward networks with stochastic outputs: infomax implies redundancy reduction. , 1998, Network.

[17]  Gustavo Deco,et al.  Unsupervised learning for blind source separation: an information-theoretic approach , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[18]  Francesco Piazza,et al.  Gradient-based blind deconvolutions with flexible approximated Bayesian estimator , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).

[19]  Francesco Piazza,et al.  Neural networks with digital LUT activation functions , 1993, Proceedings of 1993 International Conference on Neural Networks (IJCNN-93-Nagoya, Japan).

[20]  Andrzej Cichocki,et al.  Robust learning algorithm for blind separation of signals , 1994 .

[21]  Shun-ichi Amari,et al.  Adaptive Online Learning Algorithms for Blind Separation: Maximum Entropy and Minimum Mutual Information , 1997, Neural Computation.

[22]  Brian D. Ripley,et al.  Pattern Recognition and Neural Networks , 1996 .

[23]  Christian Jutten,et al.  Entropy Optimization - Application to Blind Source Separation , 1997, ICANN.

[24]  Erkki Oja,et al.  Independent component analysis by general nonlinear Hebbian-like learning rules , 1998, Signal Process..

[25]  Andrzej Cichocki,et al.  Stability Analysis of Learning Algorithms for Blind Source Separation , 1997, Neural Networks.

[26]  Andrzej Cichocki,et al.  New learning algorithm for blind separation of sources , 1992 .

[27]  Colin Fyfe,et al.  Kurtosis extrema and identification of independent components: a neural network approach , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[28]  Eric Moreau,et al.  High order contrasts for self-adaptive source separation criteria for complex source separation , 1996 .

[29]  Simone Fiori,et al.  A Study on Functional-Link Neural Units with Maximum Entropy Response , 1998 .

[30]  Juha Karhunen,et al.  Neural networks for blind separation with unknown number of sources , 1999, Neurocomputing.

[31]  Shun-ichi Amari,et al.  Learned parametric mixture based ICA algorithm , 1998, Neurocomputing.

[32]  S. Fiori.  Blind source separation by new M-WARP algorithm , 1999 .

[33]  Mark D. Plumbley.  Efficient information transfer and anti-Hebbian neural networks , 1993, Neural Networks.

[34]  A. Uncini Piazza,et al.  Artificial neural networks with adaptive polynomial activation function , 1992 .

[35]  Mohamad H. Hassoun,et al.  Nonlinear Hebbian rule: a statistical interpretation , 1994, Proceedings of 1994 IEEE International Conference on Neural Networks (ICNN'94).

[36]  Pierre Comon,et al.  Independent component analysis, a survey of some algebraic methods , 1996, 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96.

[37]  Nathalie Delfosse,et al.  Adaptive blind separation of independent sources: A deflation approach , 1995, Signal Process..

[38]  E. Oja,et al.  Independent Component Analysis , 2013 .

[39]  George Havas,et al.  A Family of Perfect Hashing Methods , 1996, Comput. J..

[40]  Barak A. Pearlmutter,et al.  Maximum Likelihood Blind Source Separation: A Context-Sensitive Generalization of ICA , 1996, NIPS.

[41]  Bartlett W. Mel,et al.  Information Processing in Dendritic Trees , 1994, Neural Computation.

[42]  Y. Bar-Ness,et al.  Bootstrap: a fast blind adaptive signal separator , 1992, [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[43]  David Horn,et al.  Probability Density Estimation Using Entropy Maximization , 1998, Neural Computation.

[44]  Erkki Oja,et al.  An Experimental Comparison of Neural Algorithms for Independent Component Analysis and Blind Separation , 1999, Int. J. Neural Syst..

[45]  Wei-Der Chang,et al.  A feedforward neural network with function shape autotuning , 1996, Neural Networks.

[46]  Francesco Piazza,et al.  Learning and Approximation Capabilities of Adaptive Spline Activation Function Neural Networks , 1998, Neural Networks.

[47]  Jean-François Cardoso,et al.  Equivariant adaptive source separation , 1996, IEEE Trans. Signal Process..

[48]  Ruey-Wen Liu,et al.  Blind signal processing: an introduction , 1996, 1996 IEEE International Symposium on Circuits and Systems. Circuits and Systems Connecting the World. ISCAS 96.

[49]  Yoram Baram,et al.  Multidimensional density shaping by sigmoids , 1996, IEEE Trans. Neural Networks.

[50]  Simone Fiori,et al.  Blind signal flattening using warping neural modules , 1998, 1998 IEEE International Joint Conference on Neural Networks Proceedings. IEEE World Congress on Computational Intelligence (Cat. No.98CH36227).