Deep learning has been proved to be one of the most effective method in feature encoding for different intelligent applications such as video-based human action recognition. However, its nonconvex optimization mechanism leads large memory consumption, which hinders its deployment on the smart embedded systems with limited computational resources. To overcome this challenge, we propose a novel deep model compression technique for smart embedded systems, which realizes both the memory size reduction and inference complexity decrease within a small drop of accuracy. First, we propose an improved naive Bayes inference-based channel parameter pruning to obtain a sparse model with higher accuracy. Then, to improve the inference efficiency, the improved Tucker decomposition method is proposed, where an improved genetic algorithm is used to optimize the Tucker ranks. Finally, to elevate the effectiveness of our proposed method, extensive experiments are conducted. The experimental results show that our method can achieve the state-of-the-art performance compared with existing methods in terms of accuracy, parameter compression, and floating-point operations reduction.