Driver distraction is one of the main causes of traffic accidents, of which two critical types are cognitive distraction and visual distraction. To improve traffic safety, the functionality of detecting driver distraction is necessary for intelligent vehicles. However, while existing studies mainly applied classification-based methods, few efforts have been devoted on modelling the relationship between input features and time dependency of driver state, which is shown to be an effective way to improve accuracy. This study proposed a linear-chain conditional random fields (CRF) based approach to detect cognitive distraction and visual distraction. Experiment was carried out on a driving simulator to collect data, where n-back task and arrow task were used to induce cognitive and visual distraction, respectively. 4 types of interpretable features were applied, including mean of skin conductance level, standard deviation of horizontal gaze position, steering reversal rates and standard deviation of lateral position. The dynamic bayesian network (DBN) used in previous studies was introduced to be the baseline. Results showed that, the proposed CRF has a superior performance than DBN, with a holistic accuracy of 93.7% and average true positive rates of 91.2% and 89.2% for cognitive distraction and visual distraction, respectively. This performance gap is due to the incorporation of input features into the transition feature functions of the designed CRF, thus making it more suitable for modelling driver state transition pattern in real application.