Pavement distress detection is of significance for road maintenance and traffic safety. Manual pavement distress detection suffers from high workloads, inefficiency, low accuracy, and high cost. To replace the manual operations in the pre-filling detection with the aim to improve efficiency and reduce cost, this paper proposes a three-stage automatic inspection and evaluation system for pavement distress based on improved deep convolutional neural networks (CNNs). First, the system integrates multi-level context information from the CNN classification model to construct discriminative super-features to determine whether there is distress in the pavement image and the type of the distress, so as to achieve rapid detection of pavement distress. Then, the pavement images with distress are fed into the CNN segmentation model to highlight the distress region with pixel-wise. In the segmentation model, a novel pyramid feature extraction module and a novel guidance attention mechanism are introduced. Finally, we evaluate the degree of pavement damage according to the segmentation results of the CNN segmentation model. In the experiments, we compare our classification model and segmentation model with other state-of-the-art methods on two pavement distress datasets, and the results demonstrate that the proposed models achieve out-performance on different evaluation metrics.