With the recent advent of low-cost acquisition depth cameras, extracting 3D body skeleton has become relatively easier, which significantly lighten many difficulties in 2D videos including occlusions, shadows and background extraction, etc. Directly perceived features, for example points, lines and planes, can be easily extracted from 3D videos such that we can employ rigid motions to represent skeletal motions in a geometric way. We apply screw matrices, acquired by using rotations and translations in 3D space, to model single and multi-body rigid motion. Since screw matrices are members of the special Euclidean group SE(3), an action can be represented as a point on a Lie group, which is a differentiable manifold. Using Lie-algebraic properties of screw algebra, isomorphic to se(3), the classical algorithms of machine learning in vector space can be expanded to manifold space. We evaluate our approached on three public 3D action datasets: MSR Action3D dataset, UCF Kinect dataset and Florence3D-Action Dataset. The experimental results show that our approaches either match or exceed state-of-the-art skeleton-based human action recognition approaches.