Deep learning detection in MIMO decode-forward relay channels

We consider a signal detection problem by using deep learning techniques in a multiple-input multiple-output (MIMO) decode-forward (DF) relay channel. There exist some suboptimal detectors such as the near maximum likelihood (NML) detector and the NML with two-level pair-wise error probability (NMLw2PEP) detector in the channel. However, the NML detectors require an exponentially increasing complexity as the number of transmit antennas increases. More seriously, without the channel state information (CSI) of the source-relay (SR) link, there is no detector that can achieve good performance even at high complexity. In this paper, we propose a deep learning approach to the NML (DL-NML) detector that achieves good performance with low complexity regardless of whether the CSI of the SR link is known or not at the destination. The DL-NML detector can detect signals in changing channels after a single training by using randomly generated channels. Furthermore, we propose a linear detector and a semidefinite relaxation approach to the NML detector to compare with the DL-NML detector in performance and complexity. The complexity analysis and simulation results validate the superiority of the proposed DL-NML detector.