New 5G technology enables network connectivity for innumerable IoT, mobile, and sensor devices. This massive involvement of tremendous devices with diverse architectures creates a complete Heterogeneous Network (HetNet) within 5G. Efficient and secure handover management in 5G HetNet is immensely challenging due to the connected devices and cell structures’ diversity. Although some researches attempt to address this issue, the complete design of the handover process, considering the diversity of 5G HetNet, is still missing. This paper introduces an SDN-enabled unified and efficient handover scheme to ensure seamless mobility in 5G HetNet. The proposed scheme expedites the performance through idle scanning and supports Device-to-Device (D2D) communication. Besides, our scheme ensures data privacy through the Elliptic Curve Integrated Encryption Scheme (ECIES) and prevents illegal access through device authentication. Security analysis endorses the security properties of the proposed scheme. Moreover, experimental results reveal that our scheme reduces handover delay by nearly 50% through idle scanning and achieves approximately 25% and 43% communication reduction in intra-domain and inter-domain handover, respectively, compared to the existing work.