A novel transceiver architecture in the two-way relay channel with analog and digital beamforming

In this paper, we propose a novel transceiver architecture at relay for multiple input, multiple output two-way relay channel (TWRC). This transceiver architecture consists of the analog beamforming in RF domain and the MMSE based digital processing at baseband, which is simplified as RF-MDP-MIMO architecture. Under the RF-MDP-MIMO architecture, the objective of this paper is to find the optimal relay transmit and receive beamformings that can achieve the boundary of the optimal capacity region with limited relay power. Specifically, by formulating the original optimization problem into two semidefinite programing (SDP) problems with rank-one constraints which can be effectively solved by the penalty function method, we can optimize relay transmit and receive beamformings alternatively. Then, the maximum achievable sum rate of TWRC can be obtained with the bisection method. Further, the achievable capacity region is realized. Numerical experiments are conducted to show that our proposed RF-MDP-MIMO architecture outperforms the RF-MIMO architecture in terms of the optimal achievable capacity region.