In this paper, a novel iterative detection technique that combines deep learning (DL) and the approximated algorithm of successive over relaxation (SOR) is proposed to achieve high reliability and reduce the computational complexity. Recently, as the demanded data rates increase, the massive multiple-input and multiple-output (MIMO) system has drawn attention in wireless communication. In massive MIMO, the implementation of traditional detectors for high reliability has become impractical, and the reduction for the complexity of detectors has emerged as a practical implementation challenge. The existing DL-based detection technique of orthogonal approximate message passing network (OAMPNet) can provide high detection performance. However, the computational complexity is too high for the implementation in massive MIMO systems. The proposed detection technique uses SOR algorithm to reduce the computational complexity, and the relaxation parameter of SOR is adaptively determined by a learning algorithm. A non-linear estimator using the DL algorithm is combined with the SOR algorithm to achieve high reliability, and regardless of the size of the MIMO system, only the size of the DL architecture determines the complexity of the non-linear estimator. Simulation results show that the proposed detector outperforms the conventional linear detector based on minimum mean square error (MMSE) and achieves high reliability with lower complexity than OAMPNet in various channel environments with spatial correlation.