H.264 is newest high performance video encoding standard that is attributed to applying several new efficient compress methods such as multiply mode inter frame encoding. There are many coding mode including different-size block mode and skip mode in inter frame coding, so it is very time-consuming and hard to use in many high demand video application. To deduce the computation complex, a fast skip mode selection algorithm based on all-zero block check is proposed. We deduce the threshold update formula first, then utilize the neighboring blocks’ motion vector to pre-judge the coding block whether appropriate for skip mode coding or not. From the experimental data, it can be see that the algorithm has extraordinary speedup ability while has little PSNR loss.