We propose a novel autonomous driving frame-work that leverages graph-based features of roads, such as road positions and connections. The proposed method is divided into two parts: a low-level controller which follows the trajectory calculated by a graph-based path planner, and a high-level controller which determines the speed of the vehicle to follow the traffic flow. The high-level controller uses a road graphical neural network (Road-GNN), which encodes a road graph into latent features to perceive the surrounding environment. We use a 3D driving simulator to test the performance of Road-GNN, which is implemented based on the satellite image data of 30 roundabout intersections. To show that the proposed method can be generalized to various road environments, the proposed method is tested using roundabouts which are different from the training set. In the experiment, the proposed method successfully trains the agent and drives an ego-vehicle through various roundabout environments. The results show that the graph-based method is effective for autonomous driving.