Non-linear transceiver design for secure communications with artificial noise-assisted MIMO relay

This study investigates the problem of physical layer security for amplify-and-forward (AF) multiple-input multiple-output (MIMO) relay systems operating in the presence of a passive eavesdropper. Specifically, the authors consider the robust design of an artificial noise (AN)-assisted non-linear transceiver employing Tomlinson–Harashima precoding (THP), with imperfect knowledge of the legitimate channel states. The design problem can be reformulated as a two-level optimisation, where the outer problem aims to optimise the source precoder as a function of the relay precoder, while the inner problem at the relay aims to jointly optimise the relay precoder as well as the power allocation between the AN and the information-bearing signals. To solve the inner problem, the authors adopt a bisection method which attempts to maximise the AN power level, to confuse the eavesdropper, while satisfying the mean-squared-error requirement for the intended user. Some relaxation for the objective function is applied to transform the problem into a standard convex optimisation one. Regarding the outer problem, closed-form solutions for the precoders can be derived by an iterative method based on the Karush–Kuhn–Tucker conditions. Simulation results illustrate the superior secrecy performance provided by the proposed non-linear transceiver design with AN and THP.