Sparse Coding and Automatic Relevance Determination for Multiway models

Multi-way modeling has become an important tool in the analysis of large scale multi-modal data. An important class of multi-way models is given by the Tucker model which decomposes the data into components pertaining to each modality as well as a core array indicating how the components of the various modalities interact. Unfortunately, the Tucker model is not unique. Furthermore, establishing the adequate model order is difficult as the number of components are specified for each mode separately. Previously, rotation criteria such as VARIMAX has been used to resolve the non-uniqueness of the Tucker representation [7]. Furthermore, all potential models have been exhaustively evaluated to estimate the adequate number of components of each mode. We demonstrate how sparse coding can prune excess components and resolve the non-uniqueness of the Tucker model while Automatic Relevance Determination in Bayesian learning form a framework to learn the adequate degree of sparsity imposed. On a wide range of multi-way data sets the proposed method is demonstrated to successfully prune excess components thereby establishing the model order. Furthermore, the non-uniqueness of the Tucker model is resolved since among potential models the models giving the sparsest representation as measured by the sparse coding regularization is attained. The approach readily generalizes to regular sparse coding as well as the CandeComp/PARAFAC model as both models are special cases of the Tucker model.

[1]  H. Kaiser.  The varimax criterion for analytic rotation in factor analysis , 1958 .

[2]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[3]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[4]  J. Kruskal.  Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics , 1977 .

[5]  Lars Nørgaard,et al.  RANK ANNIHILATION FACTOR ANALYSIS APPLIED TO FLOW INJECTION ANALYSIS WITH PHOTODIODE-ARRAY DETECTION , 1994 .

[6]  David J. Field,et al.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images , 1996, Nature.

[7]  R. Bro.  PARAFAC. Tutorial and applications , 1997 .

[8]  H. Kiers.  Joint Orthomax Rotation of the Core and Component Matrices Resulting from Three-mode Principal Components Analysis , 1998 .

[9]  Rasmus Bro,et al.  Calibration methods for complex second-order data , 1999 .

[10]  R. Bro.  Exploratory study of sugar production using fluorescence spectroscopy and multi-way analysis , 1999 .

[11]  Joos Vandewalle,et al.  A Multilinear Singular Value Decomposition , 2000, SIAM J. Matrix Anal. Appl..

[12]  Rasmus Bro,et al.  Jack-knife technique for outlier detection and estimation of standard errors in PARAFAC models , 2003 .

[13]  Bruno A Olshausen,et al.  Sparse coding of sensory inputs , 2004, Current Opinion in Neurobiology.

[14]  Yuan Qi,et al.  Predictive automatic relevance determination by expectation propagation , 2004, ICML.

[15]  D. Donoho.  For most large underdetermined systems of equations, the minimal 𝓁1‐norm near‐solution approximates the sparsest near‐solution , 2006 .

[16]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[17]  Rajat Raina,et al.  Efficient sparse coding algorithms , 2006, NIPS.

[18]  Morten Mørup,et al.  Decomposition methods for unsupervised learning , 2008 .

[19]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[20]  L. K. Hansen,et al.  Automatic relevance determination for multi‐way models , 2009 .

[21]  Feng Qianjin,et al.  Projected gradient methods for Non-negative Matrix Factorization based relevance feedback algorithm in medical image retrieval , 2011 .