Tensor-on-Tensor Regression

ABSTRACT I propose a framework for the linear prediction of a multiway array (i.e., a tensor) from another multiway array of arbitrary dimension, using the contracted tensor product. This framework generalizes several existing approaches, including methods to predict a scalar outcome from a tensor, a matrix from a matrix, or a tensor from a scalar. I describe an approach that exploits the multiway structure of both the predictors and the outcomes by restricting the coefficients to have reduced PARAFAC/CANDECOMP rank. I propose a general and efficient algorithm for penalized least-squares estimation, which allows for a ridge (L2) penalty on the coefficients. The objective is shown to give the mode of a Bayesian posterior, which motivates a Gibbs sampling algorithm for inference. I illustrate the approach with an application to facial image data. An R package is available at https://github.com/lockEF/MultiwayRegression.

[1]  Tamara G. Kolda,et al.  Tensor Decompositions and Applications , 2009, SIAM Rev..

[2]  Shmuel Friedland,et al.  Nuclear norm of higher-order tensors , 2014, Math. Comput..

[3]  Lynn E Eberly,et al.  Discriminating sample groups with multi‐way data , 2016, Biostatistics.

[4]  Demetri Terzopoulos,et al.  Multilinear Analysis of Image Ensembles: TensorFaces , 2002, ECCV.

[5]  Tal Hassner,et al.  Effective face frontalization in unconstrained images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  A. Singleton,et al.  Genetic variability in the regulation of gene expression in ten regions of the human brain , 2014, Nature Neuroscience.

[7]  Elizabeth B. Liddle,et al.  Motion-related artefacts in EEG predict neuronally plausible patterns of activation in fMRI data , 2012, NeuroImage.

[8]  A. Izenman.  Reduced-rank regression for the multivariate linear model , 1975 .

[9]  Sheng Luo,et al.  Population Value Decomposition, a Framework for the Analysis of Image Populations , 2011, Journal of the American Statistical Association.

[10]  René J. Huster,et al.  Methods for Simultaneous EEG-fMRI: An Introductory Review , 2012, The Journal of Neuroscience.

[11]  Peter D. Hoff,et al.  Separable covariance arrays via the Tucker product, with applications to multivariate relational data , 2010, 1008.2169.

[12]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[13]  Masashi Sugiyama,et al.  Theoretical and Experimental Analyses of Tensor-Based Regression and Classification , 2015, Neural Computation.

[14]  Xin Zhang,et al.  Parsimonious Tensor Response Regression , 2015, 1501.07815.

[15]  Weiwei Guo,et al.  Tensor Learning for Regression , 2012, IEEE Transactions on Image Processing.

[16]  Shree K. Nayar,et al.  Attribute and simile classifiers for face verification , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[17]  Richard A. Harshman,et al.  Foundations of the PARAFAC procedure: Models and conditions for an "explanatory" multi-model factor analysis , 1970 .

[18]  Tamara G. Kolda,et al.  Categories and Subject Descriptors: G.4 [Mathematics of Computing]: Mathematical Software— , 2022 .

[19]  Xuelong Li,et al.  Supervised tensor learning , 2005, Fifth IEEE International Conference on Data Mining (ICDM'05).

[20]  E. Lock,et al.  Supervised multiway factorization. , 2016, Electronic journal of statistics.

[21]  A. Nobel,et al.  Comment , 2011 .

[22]  Shuzhong Zhang,et al.  Maximum Block Improvement and Polynomial Optimization , 2012, SIAM J. Optim..

[23]  Jun S. Liu,et al.  The Genotype-Tissue Expression (GTEx) pilot analysis: Multitissue gene regulation in humans , 2015, Science.

[24]  L Sirovich,et al.  Low-dimensional procedure for the characterization of human faces. , 1987, Journal of the Optical Society of America. A, Optics and image science.

[25]  Shmuel Friedland,et al.  Computational Complexity of Tensor Nuclear Norm , 2014, ArXiv.

[26]  Gang Hua,et al.  Labeled Faces in the Wild: A Survey , 2016 .

[27]  Douglas W. Nychka,et al.  Tools for Spatial Data , 2016 .

[28]  R. Dennis Cook,et al.  Foundations for Envelope Models and Methods , 2015 .

[29]  Hongtu Zhu,et al.  TPRM: TENSOR PARTITION REGRESSION MODELS WITH APPLICATIONS IN IMAGING BIOMARKER DETECTION. , 2015, The annals of applied statistics.

[30]  Ji Zhu,et al.  Reduced rank ridge regression and its kernel extensions , 2011, Stat. Anal. Data Min..

[31]  Nick C Fox,et al.  The Alzheimer's disease neuroimaging initiative (ADNI): MRI methods , 2008, Journal of magnetic resonance imaging : JMRI.

[32]  L. Tucker,et al.  Some mathematical notes on three-mode factor analysis , 1966, Psychometrika.

[33]  Rainer Goebel,et al.  Predicting EEG single trial responses with simultaneous fMRI and Relevance Vector Machine regression , 2011, NeuroImage.

[34]  Hongtu Zhu,et al.  Tensor Regression with Applications in Neuroimaging Data Analysis , 2012, Journal of the American Statistical Association.

[35]  W. Sun,et al.  Sparse Low-rank Tensor Response Regression , 2016 .

[36]  C. Jack,et al.  Alzheimer's Disease Neuroimaging Initiative , 2008 .

[37]  N. G. Best,et al.  The deviance information criterion: 12 years on , 2014 .

[38]  Arjun K. Gupta,et al.  Array Variate Random Variables with Multiway Kro- necker Delta Covariance Matrix Structure , 2011 .

[39]  M. Yuan,et al.  Convex Regularization for High-Dimensional Tensor Regression , 2015 .

[40]  Lexin Li,et al.  STORE: Sparse Tensor Response Regression and Neuroimaging Analysis , 2016, J. Mach. Learn. Res..

[41]  D. Lindley,et al.  Bayes Estimates for the Linear Model , 1972 .

[42]  Deniz Akdemir.  Array Variate Skew Normal Random Variables with Multiway Kronecker Delta Covariance Matrix Structure , 2011 .

[43]  Xiaoshan Li,et al.  Tucker Tensor Regression and Neuroimaging Analysis , 2018, Statistics in Biosciences.

[44]  Dinggang Shen,et al.  TENSOR GENERALIZED ESTIMATING EQUATIONS FOR LONGITUDINAL IMAGING ANALYSIS. , 2014, Statistica Sinica.

[45]  Arthur E. Hoerl,et al.  Ridge Regression: Biased Estimation for Nonorthogonal Problems , 2000, Technometrics.

[46]  Peter D. Hoff,et al.  MULTILINEAR TENSOR REGRESSION FOR LONGITUDINAL RELATIONAL DATA. , 2014, The annals of applied statistics.

[47]  R. Sundberg.  Continuum Regression and Ridge Regression , 1993 .

[48]  N. Sidiropoulos,et al.  On the uniqueness of multilinear decomposition of N‐way arrays , 2000 .

[49]  Seungjin Choi,et al.  Color Face Tensor Factorization and Slicing for Illumination-Robust Recognition , 2007, ICB.