Secure beamforming for MIMO two-way transmission with an untrusted relay

From security perspective, a friendly relay may help to keep the confidential messages from being eavesdropped, while an untrusted relay may intentionally eavesdrop the messages when relaying. This paper studies the secure beamforming for multiple-input multiple-output (MIMO) two-way communications, where two source nodes exchange information with the help of an untrusted relay node. The relay adopts amplify-and-forward (AF) strategy and acts as both an essential helper and a potential eavesdropper. Our goal is to maximize the secrecy sum rate of the bidirectional links by jointly optimizing the source and relay beamformers. For the two-phase two-way relay scheme, we first derive the optimal structure of the relay beamformer and then propose an iterative algorithm to jointly optimize the source and relay beamformers. Then, a comprehensive study on the asymptotical performance is conducted by letting the source and relay powers approach zero or infinity. In particular, we show that when all powers approach infinity, the two-way relay scheme achieves the maximum secrecy rate if the transceiver beamformers are designed such that the received signals at the relay can be aligned to be parallel.