Mental fatigue detection is important in brain-computer interface systems (BCIs). However, owing to the time-variability and nonlinear characteristics of Electroencephalogram (EEG) signals, it is difficult to extract corresponding fatigue features of EEG signals. This paper proposes an improved CNN model based on time-frequency domain fused features which can detect mental state in BCIs. To be specific, a private EEG dataset of 8 subjects with two mental states (alertness and fatigue) induced by the 2-back task was applied. Then, 3 kinds of time domain feature-Hjorth Parameter-activity, Hjorth-mobility, Hjorth-complexity and 4 kinds of frequency domain feature: power spectral density (PSD)-a, PSD-β, PSD-θ, PSD-γ were extracted and fused together. Finally, the extracted fused features were fed into a 3 convolution layers’ convolutional neural network (CNN) model to distinguish mental states automatically. Compared with other methods for mental state classification, the proposed method achieved an average accuracy of 92.8% (max 97.8%, min 88.3%), outperforming 11 conventional methods, which indicated that the proposed method is effective for mental fatigue detection in BCIs.