Minimum Symbol Error Rate Transmitter and Receiver FIR MIMO Filters for Multilevel PSK Signaling

A theory is developed for minimizing the exact symbol error rate (SER) between the desired and decoded signals with respect to the coefficients of the transmitter and receiver finite impulse response (FIR) multiple-input multiple-output (MIMO) filters. The original source signal is assumed to be a vector source with equally likely independent identically distributed multilevel phase-shift keying (PSK) vector components. The channel is assumed to be known in both the transmitter and receiver, it is modeled as a known FIR MIMO transfer function with given order, and it is contaminated by additive circularly symmetric complex Gaussian signal-independent noise. The channel input vector signal is assumed to be power constrained. The transmitter and receiver are modeled as causal FIR MIMO filters. In the receiver, memoryless component-wise quantization is performed after the FIR MIMO receiver filter. The proposed exact SER performance expressions arc verified by Monte Carlo simulations. A numerical algorithm is proposed for jointly minimizing the SER subject to the power constraint with respect to the coefficients in the FIR transmitter and receiver FIR MIMO filters. The proposed exact SER expression reduces to known expressions from the literature for the trivial case of a scalar memoryless communication system, for zero-forcing (ZF) FIR MIMO filters with PSK signaling, and for the case of FIR MIMO filters with BPSK signaling. It is shown numerically, that the proposed method gives the same performance as the corresponding minimum mean square error (MSE) optimized system for bad channel conditions, defined as E_b/N₀. However, for good channel conditions, significantly better results than the corresponding minimum MSE optimized system are achieved. For example, for 4-PSK when SER = 10^-5, the gain in E_b/N₀ is 16.5 dB for the proposed system compared to the MSE optimized system using the exact same filter orders