Deep Learning Detection on Multi-Antenna Quantize-Forward Relay Channel Based on Maximum Likelihood

This paper investigates a deep learning (DL)-based detection scheme for phase shift keying in half-duplex multiple-input multiple-output (MIMO) quantize-forward (QF) relay channels, where the relay quantizes only the phases of received signals and forwards the phase information to the destination. While the maximum likelihood (ML) detection is theoretically optimal, its high computational complexity poses challenges for its direct application in deep learning contexts. To address this limitation, we propose an approximate ML (AML) detection method that circumvents integral calculations and adopts approximations tailored for high signal-to-noise ratio scenarios, thereby significantly reducing computational complexity. Additionally, we introduce an effective variable, λ, to mitigate the effects of small quantization bits, enabling the improved AML detection to closely match the performance of the ML detector. This modification allows the adjusted metric to better facilitate the gradient descent method for deep learning. Building on this foundation, we develop an efficient DL detection framework that achieves lower complexity and robust performance, particularly in time-varying fading channels. Furthermore, the proposed AML-λ and DL detection methods in the QF relay channel are applied to the quadrature amplitude modulation case, providing a practical and efficient solution for MIMO relay systems operating under limited memory conditions.