Background and Objective Electrocardiogram (ECG) quality assessment is significant for automatic diagnosis of cardiovascular disease and reducing the massive workload of reviewing continuous ECGs. Hence, how to design an appropriate algorithm for objectively evaluating the multi-lead ECG recordings is particularly important. Despite the deep learning methods performing well in many fields, as a data-driven method, it may not be entirely suitable for ECG analysis due to the difficulty in obtaining sufficient data and the low signal-to-noise ratio of ECG recordings. In this study, with the aim of providing an accurate and automatic ECG quality assessment scheme, we propose an innovative ECG quality assessment algorithm based on hand-crafted statistical features and deep-learned spectral features. Methods In this paper, a novel approach, combining the deep-learned Stockwell transform (S-Transform) spectrogram features and hand-crafted statistical features, is proposed for ECG quality assessment. Firstly, a double-input convolutional neural network (CNN) is established. Then, the S-Transform with a novel online augmentation scheme is performed on the multi-lead raw ECG signal received from one input layer to obtain proper time-frequency representation. After that, the CNN with three convolutional layers is employed to extract robust deep-learned features automatically. Simultaneously, the hand-crafted statistical features, including lead-fall, baseline drift, and R peak features, are calculated and fed into another input layer for feature fusion training. Finally, the deep-learned and hand-crafted features are concatenated and further fused by a fully connected layer for quality classification. Furthermore, a log-odds analysis scheme combining with a gradient-based method can localize the abnormal zone in time, frequency, and spatial domains. Results and Conclusion Our proposed method is evaluated on a publicly available database with 10-fold cross-validation. The experimental results demonstrate that the proposed assessment algorithm reached a mean accuracy of 93.09%, a mean F1-score of 0.8472, and a sensitivity of 0.9767. Moreover, comprehensive experiments indicate that the fusion of CNN features and statistical features has complementary advantages and ideal interpretability, achieving end-to-end multi-lead ECG assessment with satisfying performance.