Performance of the GLRT for adaptive vector subspace detection

The problem of adaptively detecting a signal confined to a given vector subspace in interference modeled as a zero-mean complex Gaussian N-vector is considered. The correlation properties of interference are not known but are estimated from a given set of secondary (or reference) vectors. The dimension of the known signal subspace is N_s, where 1⩽N_s⩽N. The Generalized Likelihood Ratio Test (GLRT) is cast in a slightly different setting to show that it belongs to a class of invariant tests. The maximal invariants for the class of invariant tests are identified and the joint probability density function of the maximal invariants under both the null hypothesis H₀ and the alternate hypothesis H₁ are derived. These expressions are used to show that for 1⩽N_s<N, there exists no uniformly most powerful invariant (UMPI) test for the given signal detection problem. Expressions for characterizing the performance of the GLRT are derived and the detection performance of this test when the signal to be detected is a random vector confined to the given vector subspace is evaluated