Infomax-ICA using Hessian-free optimization

We present HF-ICA, a second-order “Hessian-free” algorithm for Infomax-ICA. Our approach achieves asymptotically quadratic convergence while retaining the memory footprint of first-order methods. Without any hyperparameter tuning, we show better convergence properties than both other approximate Newton-type methods and finely-tuned stochastic Natural Gradient Descent on EEG and fMRI data. A portable, multi-threaded and vectorized C++ implementation is made publicly available along with MATLAB and Python interfaces.

[1]  Jorge Nocedal,et al.  Sample size selection in optimization methods for machine learning , 2012, Math. Program..

[2]  Seungjin Choi,et al.  A relative trust-region algorithm for independent component analysis , 2007, Neurocomputing.

[3]  Andrew W. Fitzgibbon,et al.  A fast natural Newton method , 2010, ICML.

[4]  Arnaud Delorme,et al.  EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis , 2004, Journal of Neuroscience Methods.

[5]  Aapo Hyvärinen,et al.  The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis , 1999, Neural Processing Letters.

[6]  Alexander M. Bronstein,et al.  Blind source separation using block-coordinate relative Newton method , 2004, Signal Process..

[7]  Bhaskar D. Rao,et al.  Newton method for the ICA mixture model , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[8]  Terrence J. Sejnowski,et al.  Independent Component Analysis Using an Extended Infomax Algorithm for Mixed Subgaussian and Supergaussian Sources , 1999, Neural Computation.

[9]  J. Cardoso Infomax and maximum likelihood for blind source separation , 1997, IEEE Signal Processing Letters.

[10]  James Martens,et al.  Deep learning via Hessian-free optimization , 2010, ICML.

[11]  Shun-ichi Amari,et al.  Natural Gradient Works Efficiently in Learning , 1998, Neural Computation.

[12]  Barak A. Pearlmutter Fast Exact Multiplication by the Hessian , 1994, Neural Computation.

[13]  O. Chapelle Improved Preconditioner for Hessian Free Optimization , 2011 .

[14]  Alexander M. Bronstein,et al.  Blind Source Separation Using the Block-Coordinate Relative Newton Method , 2004, ICA.

[15]  Terrence J. Sejnowski,et al.  Blind separation and blind deconvolution: an information-theoretic approach , 1995, 1995 International Conference on Acoustics, Speech, and Signal Processing.