The tendency towards carbon dioxide reduction greatly stimulates the popularity of electric vehicles against conventional vehicles. However, electric vehicle chargers represent a huge electric burden, which affects the performance and stability of the grid. Various optimization methodologies have been proposed in literature to enhance the performance of the distribution grids. However, existing techniques handle the raised issues from individual perspectives and/or with limited scopes. Therefore, this paper aims to develop a distributed controller-based coordination scheme in both medium and low voltage networks to handle the electric vehicles’ charging impact on the power grid. The scope of this work covers improving the network voltage profile, reducing the total active and reactive power, reducing the load fluctuations and total charging cost, while taking into consideration the random arrivals/departures of electric vehicles and the vehicle owners’ preferred charging time zones with vehicle-to-grid technology. Simulations are carried out to prove the success of the proposed method in improving the performance of IEEE 31-bus 23 kV system with several 415 V residential feeders. Additionally, the proposed method is validated using Controller Hardware-in-the-Loop. The results show that the proposed method can significantly reduce the issues that appear in the electric power grid during charging with minor changes in the existing grid. The results prove the successful implementation of different types of charging, namely, ultra-fast, fast, moderate, normal and vehicle-to-grid charging with minimum charging cost to enhance the owner’s satisfaction level.