We design and implement an autonomous driving system based on image recognition using programmable SoCs. The proposed system equips two FPGA boards and three cameras. One FPGA board implements a driving control system, and the other FPGA board implements object detection and recognition using machine learning algorithms. Driving control is performed based on road edge line detection and road marking recognition using the canny edge detection. On the other hand, image detection and recognition of traffic lights are implemented using the random forest method with HOG features. In the development framework of programmable SoC of Zynq 7000, we adopt a Hardware/Software co-design to balance the design period and system performance required for real-time processing.