In this paper, we present an action recognition framework based on binary stochastic latent variables model, Hidden unit Conditional Random Fields(HuCRF). It is a chain structured undirected graphs model with nonlinear dependencies at each frame/segment, contrast to standard log-linear models like CRF. So it is more powerful in sequence modeling tasks like action recognition. The observations of actions can be various and multi-cues. In this paper, we focus on (but not limited to) indoor daily life action surveillance, and the raw data are collected by RGBD sensors(Microsoft Kinect), including RGBD videos and skeleton data. The experiments results on benchmark Datasets show that our model performs well in the action recognition task.