Analysis of per-layer transmit and receive filters design for multiuser MIMO systems with Tomlinson-Harashima precoding

In this paper, the per-layer design for Tomlinson-Harashima precoding (THP) in the downlink of multiuser multiple-input multiple-output (MIMO) systems is investigated. In these systems, multiple antennas are located at all the transceivers, and the number of the receivers is equal to that of the transmit antennas. Based on the criterion of maximizing system sum-capacity, we propose two per-layer joint transmit and receive filters design schemes with receive antenna beamforming (RAB) and receive antenna selection (RAS), respectively. Moreover, the differences of the equivalent channel gains between these two schemes are theoretically analyzed. Simulation results indicate that by the proposed schemes, the system sum-rate can be improved significantly relative to the per-user processing scheme.