Limiting spectral distribution of renormalized separable sample covariance matrices when

We are concerned with the behavior of the eigenvalues of renormalized sample covariance matrices of the form C n = n p ( 1 n A p 1 / 2 X n B n X n ∗ A p 1 / 2 − 1 n tr ( B n ) A p ) as p , n → ∞ and p / n → 0 , where X n is a p × n matrix with i.i.d. real or complex valued entries X i j satisfying E ( X i j ) = 0 , E | X i j | 2 = 1 and having finite fourth moment. A p 1 / 2 is a square-root of the nonnegative definite Hermitian matrix A p , and B n is an n × n nonnegative definite Hermitian matrix. We show that the empirical spectral distribution (ESD) of C n converges a.s. to a nonrandom limiting distribution under the assumption that the ESD of A p converges to a distribution F A that is not degenerate at zero, and that the first and second spectral moments of B n converge. The probability density function of the LSD of C n is derived and it is shown that it depends on the LSD of A p and the limiting value of n − 1 tr ( B n 2 ) . We propose a computational algorithm for evaluating this limiting density when the LSD of A p is a mixture of point masses. In addition, when the entries of X n are sub-Gaussian, we derive the limiting empirical distribution of { n / p ( λ j ( S n ) − n − 1 tr ( B n ) λ j ( A p ) ) } j = 1 p where S n ≔ n − 1 A p 1 / 2 X n B n X n ∗ A p 1 / 2 is the sample covariance matrix and λ j denotes the j th largest eigenvalue, when F A is a finite mixture of point masses. These results are utilized to propose a test for the covariance structure of the data where the null hypothesis is that the joint covariance matrix is of the form A p ⊗ B n for ⊗ denoting the Kronecker product, as well as A p and the first two spectral moments of B n are specified. The performance of this test is illustrated through a simulation study.