Enhancing physical layer security in untrusted relay networks with artificial noise: A symbol error rate based approach

We apply the concept of artificial noise (AN) transmission in an untrusted relay network to enhance secure communication between the source and destination. Specifically, in addition to square quadrature amplitude modulated (QAM) signals broadcasted by the source, the relay simultaneously receives AN symbols designed and transmitted by the destination. For the relay, based on the assumption of additive white Gaussian noise, we derive the analytical symbol error rate (SER) expression, which is maximized for the optimal AN design. Under an average power constraint, we find the optimal phase and power distribution of the AN. Interestingly, our study shows that the Gaussian distribution is generally not optimal to generate AN and the results in this paper can be used as benchmarks for future analyses of AN-based techniques. More importantly, rather than conducting analysis from an information-theoretic perspective, our SER-based approach takes practical communication issues into account, such as QAM signalling.