Random matrix theory applied to low rank stap detection

The paper addresses the problem of target detection embedded in a disturbance composed of a low rank Gaussian clutter and a white Gaussian noise. In this context, it is interesting to use an adaptive version of the Low Rank Normalized Matched Filter detector, denoted LR-ANMF, which is a function of the estimation of the projector onto the clutter subspace. In this paper, we show that the LR-ANMF detector based on the sample covariance matrix is consistent when the number of secondary data K tends to infinity for a fixed data dimension m but not consistent when m and K both tend to infinity at the same rate, i.e. m/K → c ∈ (0, ∞). Using the results of random matrix theory, we then propose a new version of the LR-ANMF which is consistent in both cases. The application of our new detector on STAP (Space Time Adaptive Processing) data shows the interest of our approach.