Rate distortion optimization (RDO) is a very efficient tool for deciding a macroblock's coding mode, which has been adopted by the H.264/AVC codec and brings higher coding efficiency. However, due to the precise calculation of the current macroblock's distortion and coding-bits for each coding mode, RDO-based mode decision involves high computational complexity. In this paper, a simple model for quickly estimating the coding-bits of the quantized transform coefficients, which account for the major part of the computation when coding a macroblock, is presented. The model only uses nonzero coefficients' magnitude and coordinates information by add operations to fulfill estimation so that computational complexity can be significantly reduced. Experimental results show that with this model the computational complexity of RDO-based mode decision in the H.264/AVC codec can be reduced by 40-60% while coding efficiency has little drop.