An optimization approach to decision feedback detection under modulation constraints for MIMO fading channels

A new technique is proposed for designing decision feedback detectors (DFDs) wherein the per-symbol decision rules are obtained by exploiting modulation constraints. It is presented in the context of the multi-input, multi-output (MIMO) fading channel with K transmit and N receive antennas. Three cases are considered to illustrate the technique where all the transmitters employ real-valued pulse-amplitude modulation (PAM), or phaseshift keying (PSK), or complex quadrature amplitude modulation (QAM). In each case, the corresponding per-symbol decision rules are obtained by imposing the modulation constraints on a likelihood function maximization problem. When all transmitters employ PAM, it is shown that the resulting DFD (the PAM-DFD) is equivalent to the decorrelating decision feedback detector (DDFD) when the latter is used on a modified received statistic. Two DFDs are then derived (the PSK-DFDs and the QAMDFDs) for the cases when all transmitters employ PSK and QAM, respectively. To illustrate the performance benefit, we consider the Rayleigh fading channel and derive the exact joint error probability (JEP) of the PAM-DFD and show that the (possibly fractional) diversity order of the JEP is equal to N - K-1/ 2 . This greatly improves on the diversity order of N - K + 1 of the JEP obtained by the D-DFD without exploiting the real modulation constraint. Through simulations, it is shown that the PSK-DFDs and the QAM-DFDs result in significant performance improvements over the D-DFD as well. It is conjectured that one of the PSK-DFD achieves the improved diversity order of N - K-1/ 2 as well and the QAM-DFDs result in an improvement in effective SNR gain.