Limiting spectral distribution of renormalized separable sample covariance matrices when p/ni → 0

We are concerned with the behavior of the eigenvalues of renormalized sample covariance matrices of the form C"n=np(1nA"p^1^/^2X"nB"nX"n^*A"p^1^/^2-1ntr(B"n)A"p) as p,n->~ and p/n->0, where X"n is a pxn matrix with i.i.d. real or complex valued entries X"i"j satisfying E(X"i"j)=0, E|X"i"j|^2=1 and having finite fourth moment. A"p^1^/^2 is a square-root of the nonnegative definite Hermitian matrix A"p, and B"n is an nxn nonnegative definite Hermitian matrix. We show that the empirical spectral distribution (ESD) of C"n converges a.s. to a nonrandom limiting distribution under the assumption that the ESD of A"p converges to a distribution F^A that is not degenerate at zero, and that the first and second spectral moments of B"n converge. The probability density function of the LSD of C"n is derived and it is shown that it depends on the LSD of A"p and the limiting value of n^-^1tr(B"n^2). We propose a computational algorithm for evaluating this limiting density when the LSD of A"p is a mixture of point masses. In addition, when the entries of X"n are sub-Gaussian, we derive the limiting empirical distribution of {n/p(@l"j(S"n)-n^-^1tr(B"n)@l"j(A"p))}"j"="1^p where S"[email protected]?n^-^1A"p^1^/^2X"nB"nX"n^*A"p^1^/^2 is the sample covariance matrix and @l"j denotes the jth largest eigenvalue, when F^A is a finite mixture of point masses. These results are utilized to propose a test for the covariance structure of the data where the null hypothesis is that the joint covariance matrix is of the form A"[email protected]?B"n for @? denoting the Kronecker product, as well as A"p and the first two spectral moments of B"n are specified. The performance of this test is illustrated through a simulation study.