On the design of ZF and MMSE Tomlinson-Harashima precoding in multiuser MIMO amplify-and-forward relay system

In this paper, we design a zero-forcing (ZF) Tomlinson-Harashima precoding (THP) and a minimum mean square error (MMSE) THP for multiuser MIMO relay system with direct link. Based on these precoding structures, we propose the optimal combining weights of the ZF THP and the MMSE THP in half duplex relay system, and the sub-optimal weights of MMSE THP to reduce feedback channel state information (CSI). Through the Monte Carlo simulation, we verify that the optimal weights combining method outperforms the constant weights combining method. In addition, we confirm that the performance of the sub-optimal weights combining method is close to the optimal method, while reducing the feedback CSI simultaneously.