Blind Signal Separation and Extraction: Recent Trends, Future Perspectives, and Applications

Blind source separation (BSS) and related methods, e.g., ICA are generally based on a wide class of unsupervised learning algorithms and they found potential applications in many areas from engineering to psychology and neuroscience. The recent trends in BSS is to consider problems in the framework of probabilistic generative and tree structured graphical models and exploit a priori knowledge about true nature and structure of latent (hidden) variables or sources such as statistical independence, spatio-temporal decorrelation, sparseness, smoothness or linear predictability. The goal of BSS can be considered as estimation of sources and parameters of a mixing system or more generally as finding a new reduced or compressed representation for the observed (sensor) data that can be interpreted as physically meaningful coding or blind source extraction. The key issue is to find a such transformation or coding (linear or nonlinear) which has true physical meaning and interpretation. In this paper, we briefly review some promising linear models and approaches to blind source separation and extraction using various criteria and assumptions.

[1]  Liqing Zhang,et al.  Semiparametric model and superefficiency in blind deconvolution , 2001, Signal Process..

[2]  Klaus Schulten,et al.  A Numerical Study on Learning Curves in Stochastic Multilayer Feedforward Networks , 1996, Neural Computation.

[3]  Yuanqing Li,et al.  Analysis of Sparse Representation and Blind Source Separation , 2004, Neural Computation.

[4]  Si Wu,et al.  Information processing in a neuron ensemble with the multiplicative correlation structure , 2004, Neural Networks.

[5]  K. Hiraoka.  Optimal Strategy under Unknown Stochastic Environment | Nonparametric Lob-Pass Problem , 1995 .

[6]  S. Amari,et al.  Information Geometry of α-Projection in Mean Field Approximation , 2004 .

[7]  Andrzej Cichocki,et al.  Independent component analysis for unaveraged single-trial MEG data decomposition and single-dipole source localization , 2002, Neurocomputing.

[8]  Andrzej Cichocki,et al.  Self-whitening algorithms for adaptive equalization and deconvolution , 1999, IEEE Trans. Signal Process..

[9]  Andrzej Cichocki,et al.  A robust approach to independent component analysis of signals with high-level noise measurements , 2003, IEEE Trans. Neural Networks.

[10]  Andrzej Cichocki,et al.  Adaptive blind signal and image processing , 2002 .

[11]  S. Amari,et al.  Flexible Independent Component Analysis , 1998, Neural Networks for Signal Processing VIII. Proceedings of the 1998 IEEE Signal Processing Society Workshop (Cat. No.98TH8378).

[12]  Magnus Rattray,et al.  Natural gradient descent for on-line learning , 1998 .

[13]  Shun-ichi Amari,et al.  Exponential Convergence of Delayed Dynamical Systems , 2001, Neural Computation.

[14]  S. Amari,et al.  Differential and Algebraic Geometry of Multilayer Perceptrons , 2001 .

[15]  Liqing Zhang,et al.  Self-adaptive blind source separation based on activation functions adaptation , 2004, IEEE Transactions on Neural Networks.

[16]  Kenji Fukumizu,et al.  Adaptive Method of Realizing Natural Gradient Learning for Multilayer Perceptrons , 2000, Neural Computation.

[17]  Jacek M. Zurada,et al.  Blind extraction of singularly mixed source signals , 2000, IEEE Trans. Neural Networks Learn. Syst..

[18]  S. Amari,et al.  Estimating Functions in Semiparametric Statistical Models , 1997 .

[19]  Shun-ichi Amari,et al.  The EM Algorithm and Information Geometry in Neural Network Learning , 1995, Neural Computation.

[20]  Shun-ichi Amari,et al.  Global Convergence Rate of Recurrently Connected Neural Networks , 2002, Neural Computation.

[21]  Shun-ichi Amari,et al.  Information geometry of the EM and em algorithms for neural networks , 1995, Neural Networks.

[22]  Shun-ichi Amari,et al.  Estimating Functions of Independent Component Analysis for Temporally Correlated Signals , 2000, Neural Computation.

[23]  Shun-ichi Amari,et al.  Information-Geometric Measure for Neural Spikes , 2002, Neural Computation.

[24]  S. Amari,et al.  Dualistic differential geometry of positive definite matrices and its applications to related problems , 1996 .

[25]  Sergio Cruces,et al.  On a new blind signal extraction algorithm: different criteria and stability analysis , 2002, IEEE Signal Processing Letters.

[26]  S. Amari,et al.  A self-stabilized minor subspace rule , 1998, IEEE Signal Processing Letters.

[27]  Sergio VerdÂ,et al.  Statistical Inference Under Multiterminal Data Compression , 2000 .

[28]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[29]  Shun-ichi Amari,et al.  Learning Coefficients of Layered Models When the True Distribution Mismatches the Singularities , 2003, Neural Computation.

[30]  Masato Inoue,et al.  Improved Parameter Estimation for Variance-stabilizing Transformation of Gene-expression Microarray Data , 2004, J. Bioinform. Comput. Biol..

[31]  S. Amari,et al.  Gradient systems in view of information geometry , 1995 .

[32]  S. Sathiya Keerthi,et al.  A fast iterative nearest point algorithm for support vector machine classifier design , 2000, IEEE Trans. Neural Networks Learn. Syst..

[33]  Shun-ichi Amari,et al.  Annealed online learning in multilayer neural networks , 1999 .

[34]  Shun-ichi Amari,et al.  Parameter estimation with multiterminal data compression , 1995, IEEE Trans. Inf. Theory.

[35]  Si Wu,et al.  Conformal Transformation of Kernel Functions: A Data-Dependent Way to Improve Support Vector Machine Classifiers , 2002, Neural Processing Letters.

[36]  Si Wu,et al.  Attention Modulation of Neural Tuning Through Peak and Base Rate , 2001, Neural Computation.

[37]  Shun-ichi Amari,et al.  Statistical Analysis of Regularization Constant - From Bayes, MDL and NIC Points of View , 1997, IWANN.

[38]  Soo-Young Lee,et al.  Adaptive noise cancelling based on independent component analysis , 2002 .

[39]  Joseph F. Murray,et al.  Dictionary Learning Algorithms for Sparse Representation , 2003, Neural Computation.

[40]  S. Amari,et al.  A Cascade Neural Network for Blind Signal Extraction without Spurious Equilibria , 1998 .

[41]  S. Amari,et al.  Information geometry of estimating functions in semi-parametric statistical models , 1997 .

[42]  Shun-ichi Amari,et al.  Convergence of the Wake-Sleep Algorithm , 1998, NIPS.

[43]  S. Amari.  [Neural Networks: A Review from Statistical Perspective]: Comment , 1994 .

[44]  Shun-ichi Amari,et al.  Statistical analysis of learning dynamics , 1999, Signal Process..

[45]  Shun-ichi Amari,et al.  Auto-associative memory with two-stage dynamics of nonmonotonic neurons , 1996, IEEE Trans. Neural Networks.

[46]  Si Wu,et al.  Improving support vector machine classifiers by modifying kernel functions , 1999, Neural Networks.

[47]  Michael I. Jordan,et al.  Beyond Independent Components: Trees and Clusters , 2003, J. Mach. Learn. Res..

[48]  Shun-ichi Amari,et al.  BSC: Testing of Hypotheses with Information Constraints , 2000 .

[49]  Nikola Kasabov,et al.  Brain-like Computing and Intelligent Information Systems , 1998 .

[50]  Kazushi Ikeda,et al.  Geometry of Admissible Parameter Region in Neural Learning , 1996 .

[51]  Shun-ichi Amari.  Independent Component Analysis (ICA) and Method of Estimating Functions , 2002 .

[52]  Shun-ichi Amari,et al.  Learning and inference in hierarchical models with singularities , 2003, Systems and Computers in Japan.

[53]  Shun-ichi Amari,et al.  Differential geometric structures of stable state feedback systems with dual connections , 1992, Kybernetika.

[54]  Shun-ichi Amari,et al.  New theorems on global convergence of some dynamical systems , 2001, Neural Networks.

[55]  Andrzej Cichocki,et al.  Stability Analysis of Learning Algorithms for Blind Source Separation , 1997, Neural Networks.

[56]  Shun-ichi Amari,et al.  Geometry of Learning in Multilayer Perceptrons , 2004 .

[57]  Shun-ichi Amari,et al.  Unified stabilization approach to principal and minor components extraction algorithms , 2001, Neural Networks.

[58]  Andrzej Cichocki,et al.  Robust techniques for independent component analysis (ICA) with noisy data , 1998, Neurocomputing.

[59]  Motoaki Kawanabe,et al.  Estimation of Network Parameters in Semiparametric Stochastic Perceptron , 1994, Neural Computation.

[60]  Ulla Ruotsalainen,et al.  Neuroinformatics: the integration of shared databases and tools towards integrative neuroscience. , 2002, Journal of integrative neuroscience.

[61]  Pando G. Georgiev,et al.  Blind Source Separation Algorithms with Matrix Constraints , 2003, IEICE Trans. Fundam. Electron. Commun. Comput. Sci..

[62]  Shun-ichi Amari,et al.  Methods of information geometry , 2000 .

[63]  S. Amari,et al.  Fast-convergence filtered regressor algorithms for blind equalisation , 1996 .

[64]  Shun-ichi Amari,et al.  Adaptive Online Learning Algorithms for Blind Separation: Maximum Entropy and Minimum Mutual Information , 1997, Neural Computation.

[65]  Shun-ichi Amari,et al.  On Some Singularities in Parameter Estimation Problems , 2003, Probl. Inf. Transm..

[66]  Kenji Fukumizu,et al.  Adaptive natural gradient learning algorithms for various stochastic models , 2000, Neural Networks.

[67]  Shun-ichi Amari,et al.  Natural Gradient Learning for Over- and Under-Complete Bases in ICA , 1999, Neural Computation.

[68]  Shun-ichi Amari,et al.  Neural Network Models for Blind Separation of Time Delayed and Convolved Signals , 1997 .

[69]  Andrzej Cichocki,et al.  A New Learning Algorithm for Blind Signal Separation , 1995, NIPS.

[70]  Si Wu,et al.  Sequential Bayesian Decoding with a Population of Neurons , 2003, Neural Computation.

[71]  Shun-ichi Amari,et al.  The Lob-Pass Problem , 2000, J. Comput. Syst. Sci..

[72]  Shun-ichi Amari,et al.  Blind source separation-semiparametric statistical approach , 1997, IEEE Trans. Signal Process..

[73]  Andreas Ziehe,et al.  On-line learning in switching and drifting environments with application to blind source separation , 1999 .

[74]  Shun-ichi Amari,et al.  Improving Generalization Performance of Natural Gradient Learning Using Optimized Regularization by NIC , 2004, Neural Computation.

[75]  E. Oja,et al.  Independent Component Analysis , 2013 .

[76]  Shun-ichi Amari,et al.  Adaptive blind signal processing-neural network approaches , 1998, Proc. IEEE.

[77]  Shun-ichi Amari,et al.  On Density Estimation under Relative Entropy Loss Criterion , 2002, Probl. Inf. Transm..

[78]  Shun-ichi Amari,et al.  Stochastic Reasoning, Free Energy, and Information Geometry , 2004, Neural Computation.

[79]  Andrzej Cichocki,et al.  Nonholonomic Orthogonal Learning Algorithms for Blind Source Separation , 2000, Neural Computation.

[80]  Klaus-Robert Müller,et al.  Asymptotic statistical theory of overtraining and cross-validation , 1997, IEEE Trans. Neural Networks.

[81]  Shun-ichi Amari,et al.  Dynamic behavior of the robust decorrelation process , 2000, Neurocomputing.

[82]  Qin Lin,et al.  A unified algorithm for principal and minor components extraction , 1998, Neural Networks.

[83]  Jacek M. Zurada,et al.  Temporal-to-spatial dynamic mapping, flexible recognition, and temporal correlations in an olfactory cortex model , 2002, Biological Cybernetics.

[84]  Tsunehiro Takeda,et al.  Single-Trial Magnetoencephalographic Data Decomposition and Localization Based on Independent Component Analysis Approach , 2000 .

[85]  Kenji Fukumizu,et al.  Local minima and plateaus in hierarchical structures of multilayer perceptrons , 2000, Neural Networks.

[86]  Rajesh P. N. Rao.  Bayesian Computation in Recurrent Neural Circuits , 2004, Neural Computation.

[87]  Barak A. Pearlmutter,et al.  Blind Source Separation via Multinode Sparse Representation , 2001, NIPS.

[88]  Cheng-Hsiung Hsieh.  Grey Filtering and Its Application to Speech Enchancement , 2003 .

[89]  Shun-ichi Amari,et al.  Information geometry on hierarchy of probability distributions , 2001, IEEE Trans. Inf. Theory.

[90]  Liqing Zhang,et al.  Multichannel blind deconvolution of nonminimum-phase systems using filter decomposition , 2004, IEEE Transactions on Signal Processing.

[91]  Shun-ichi Amari,et al.  Strategy Under the Unknown Stochastic Environment: the Nonparametric Lob—Pass Problem , 1998, Algorithmica.

[92]  Si Wu,et al.  Asymptotic behaviors of population codes , 2002, Neurocomputing.

[93]  Shun-ichi Amari,et al.  Complexity Issues in Natural Gradient Descent Method for Training Multilayer Perceptrons , 1998, Neural Computation.

[94]  J. S. Lee,et al.  Non-negative matrix factorization of dynamic images in nuclear medicine , 2001, IEEE Nuclear Science Symposium Conference Record.

[95]  Andrzej Cichocki,et al.  A common neural-network model for unsupervised exploratory data analysis and independent component analysis , 1998, IEEE Trans. Neural Networks.

[96]  Klaus-Robert Müller,et al.  Statistical Theory of Overtraining - Is Cross-Validation Asymptotically Effective? , 1995, NIPS.

[97]  Shun-ichi Amari,et al.  Sequential Extraction of Minor Components , 2001, Neural Processing Letters.

[98]  S. Amari,et al.  Statistical inference: learning in artificial neural networks , 1998, Trends in Cognitive Sciences.

[99]  M. V. Burnashev,et al.  On Some Estimation Problems with Information Constraints , 2002 .

[100]  Shun-ichi Amari,et al.  Stability of asymmetric Hopfield networks , 2001, IEEE Trans. Neural Networks.

[101]  Si Wu,et al.  Population Coding and Decoding in a Neural Field: A Computational Study , 2002, Neural Computation.

[102]  Shun-ichi Amari,et al.  Information geometry of neural network—an overview , 1997 .

[103]  Jacek M. Zurada,et al.  Nonlinear Blind Source Separation Using a Radial Basis Function Network , 2001 .

[104]  최승진.  Approximate maximum likelihood source separation using the natural gradient , 2001 .

[105]  Andrzej Cichocki,et al.  Blind source separation with convolutive noise cancellation , 1997, Neural Computing & Applications.

[106]  Sergio Cruces,et al.  From blind signal extraction to blind instantaneous signal separation: criteria, algorithms, and stability , 2004, IEEE Transactions on Neural Networks.

[107]  Liqing Zhang,et al.  Natural gradient algorithm for blind separation of overdetermined mixture with additive noise , 1999, IEEE Signal Processing Letters.

[108]  Masato Inoue,et al.  Gene Interaction in DNA Microarray Data Is Decomposed by Information Geometric Measure , 2003, Bioinform..

[109]  Shun-ichi Amari,et al.  Sequential blind signal extraction in order specified by stochastic properties , 1997 .

[110]  Shun-ichi Amari,et al.  Self-Organization in the Basal Ganglia with Modulation of Reinforcement Signals , 2002, Neural Computation.

[111]  Shun-ichi Amari,et al.  Information geometry of turbo and low-density parity-check codes , 2004, IEEE Transactions on Information Theory.

[112]  Andrzej Cichocki,et al.  Equivariant Nonstationary Source Separation , 2002 .

[113]  Shun-ichi Amari,et al.  Attention modulation of neural tuning through peak and base rate in correlated firing , 2002, Neural Networks.

[114]  Ulla Ruotsalainen,et al.  Neuroscience data and tool sharing , 2003, Neuroinformatics.

[115]  Andrzej Cichocki,et al.  Bias removal technique for blind source separation with noisy measurements , 1998 .

[116]  Shun-ichi Amari,et al.  Superefficiency in blind source separation , 1999, IEEE Trans. Signal Process..

[117]  Andrzej Cichocki,et al.  Information-theoretic approach to blind separation of sources in non-linear mixture , 1998, Signal Process..

[118]  Si Wu,et al.  Population Coding with Correlation and an Unfaithful Model , 2001, Neural Computation.

[119]  Shun-ichi Amari,et al.  Mutual information of sparsely coded associative memory with self-control and ternary neurons , 2000, Neural Networks.

[120]  Yutaka Sakai,et al.  Synchronous Firing and Higher-Order Interactions in Neuron Pool , 2003, Neural Computation.