Joint transmitter/receiver optimization for overlay wireless systems

In wireless communication paradigms such as cognitive radio, the overlay user operates only when the channel is idle. We propose a framework that allows simultaneous co-channel operation of both the legacy and the overlay user. The sum of the mean-squared error (MSE) of the new system plus the excess MSE in the existing system due to the introduction of the overlay system is minimized by designing the transmitter and the receiver of the overlay system. Simulation results under fading condition show that our proposed method can yield significant improvement in the system performance by achieving a gain of 10dB compared to an un-optimized system.