The smart grid is generally studied as an efficient and powerful electric grid. With the assistance of information and communication technology (ICT), the electric grid can increase the performance of the power grid system with smart energy management. On the other hand, with the usage of renewable energy resources (RERs), smart energy storage, and new transmission technologies in the power grid system, various new features such as real-time monitoring, fast restoration, battery displays, automated outage management, etc. have been assimilated into the smart grid. These new features generate more complexity in energy transmission and constitute important challenges like low energy consumption, high energy cost, social welfare, etc. while designing energy trading mechanisms in the smart grid. In the Internet-of-Things (IoT) era, several scenarios such as micro-grids, energy harvesting networks, and vehicle-to-grid (V2G) networks are present where energy trading plays an important role. However, in these scenarios, there are energy transmission and distribution, security and privacy, energy consumption, system reliability, the criticality of data delivery, and a few more challenges caused by distrust, non-transparent, and uncertain energy markets. Motivated from these challenges, we present a four-layered architecture of energy trading used in the smart grid. We propose a comprehensive background regarding the main concepts of energy trading and the implication of enabling technologies that manage the energy imbalances in the smart grid. Then, we present a problem taxonomy based on incentive, mathematical, and simulation model-driven approaches, which are widely used to control and maintain the energy trading mechanisms. Based on the findings from the literature, we also present a solution taxonomy with enabling technologies such as Energy Internet, Software-defined networking (SDN), and blockchain. In the end, a summary of future research directions based on the energy trading mechanisms is explored to provide deep insights to the readers.