Joint MMSE precoding design in multi-user two-way MIMO relay systems

This paper addresses the problem of joint design of user precoders and receivers as well as the relay precoder based on the minimum mean-square-error (MMSE) criterion in the multi-user amplify-and-forward (AF) two-way multiple-input multiple-output (MIMO) relay systems with analog network coding (ANC). By decoupling the original highly nonconvex problem into three sub-problems, we propose an iterative algorithm to repeatedly optimize precoders and receivers. The convergence of the algorithm is ensured since the updated solution is optimal to each sub-problem. In addition, a novel channel switching technique at the relay node is implemented as means of exploring different local optimal solutions to further improve the system performance. Numerical simulation results demonstrate that the proposed iterative algorithm outperforms other algorithms in terms of both bit error rate (BER) and mean square error (MSE).