Best rank-k approximations for tensors: generalizing Eckart–Young

Given a tensor f in a Euclidean tensor space, we are interested in the critical points of the distance function from f to the set of tensors of rank at most k, which we call the critical rank-at-most-k tensors for f. When f is a matrix, the critical rank-one matrices for f correspond to the singular pairs of f. The critical rank-one tensors for f lie in a linear subspace $$H_f$$Hf, the critical space of f. Our main result is that, for any k, the critical rank-at-most-k tensors for a sufficiently general f also lie in the critical space $$H_f$$Hf. This is the part of Eckart–Young Theorem that generalizes from matrices to tensors. Moreover, we show that when the tensor format satisfies the triangle inequalities, the critical space $$H_f$$Hf is spanned by the complex critical rank-one tensors. Since f itself belongs to $$H_f$$Hf, we deduce that also f itself is a linear combination of its critical rank-one tensors.