A Design of Optimal Overlay System Inducing Zero Interference to Legacy Systems

In this paper, the jointly optimal transmitter and receiver pair of an overlay system is derived to minimize the mean squared error (MSE) at the output of the linear receiver, subject to zero interference induced to legacy systems already existing in the frequency band of interest. It is assumed that the overlay and the legacy systems all employ linear modulation with the same symbol transmission rate but they are not necessarily symbol- synchronized. The symbol sequences are modeled as wide-sense stationary (WSS) colored random processes and the channels are modeled as strictly band-limited linear time-invariant systems with frequency selective impulse responses. Using the vectorized Fourier transform (VFT) technique, the optimal transmit and receive waveforms are derived in the frequency domain. Numerical results show that the proposed system significantly improves the bit error rate (BER) performance over the systems with receiver-only optimization and the systems with no transmitter and receiver optimization.