The dense disparity map generation algorithm based on segmentation solves, to a large extent, the problems of noise or non-disparity values area brought by the algorithm based on pixel, while at the same time it may generate the wrong disparity values. In the traditional algorithm based on pixel, only the color information is taken into consideration, so the different adjacent planes with consistent color are segmented into the same area, and then the wrong plane fitting of disparity values is generated. Thus, a dense disparity map generation method based on color and disparity gradient is proposed in this paper. At first, the directional derivative of the initial disparity map is computed, and the disparity gradient amplitude and disparity gradient angle are computed with the directional derivative. The information of color, disparity gradient amplitude and disparity gradient angle is combined as the new segmentation characteristics to segment the image. Then, according to the result of segmentation and initial disparity map, the disparity values are computed through plane fitting, and the final dense disparity map is obtained. The experimental results show that this algorithm can better solve the problem of wrong segmentation of the areas with similar color but on different planes, and that then the more accurate dense disparity map can be obtained.