Improving and characterizing the performance of stochastic matched subspace detectors when using noisy estimated subspaces

We consider a stochastic matched subspace detection problem where the signal subspace is unknown and estimated by taking the eigenvalue decomposition of the sample covariance matrix of noisy signal-bearing training data. In moderate to low signal-to-noise ratio (SNR) regimes or in the setting where the number of samples is limited, subspace estimation errors affect the performance of matched subspace detectors. We use random matrix theory to derive an optimal matched subspace detector which accounts for these estimation errors and to analytically predict the associated ROC performance curves. What emerges from the analysis is the importance of using only the k_eff ≤ k informative signal subspace components that can be reliably estimated from the noisy, limited data. Specifically, the ROC analysis shows that the performance of the optimal detector matches that of the plug-in detector that uses exactly k_eff components. The analytical predictions are validated using numerical simulations.