Building boundary optimization is an essential post-process step for building extraction (by image classification). However, current boundary optimization methods through smoothing or line fitting principles are unable to optimize complex buildings. In response to this limitation, this paper proposes an object-oriented building contour optimization method via an improved generalized gradient vector flow (GGVF) snake model and based on the initial building contour results obtained by a classification method. First, to reduce interference from the adjacent non-building object, each building object is clipped via their extended minimum bounding rectangles (MBR). Second, an adaptive threshold Canny edge detection is applied to each building image to detect the edges, and the progressive probabilistic Hough transform (PPHT) is applied to the edge result to extract the line segments. For those cases with missing or wrong line segments in some edges, a hierarchical line segments reconstruction method is designed to obtain complete contour constraint segments. Third, accurate contour constraint segments for the GGVF snake model are designed to quickly find the target contour. With the help of the initial contour and constraint edge map for GGVF, a GGVF force field computation is executed, and the related optimization principle can be applied to complex buildings. Experimental results validate the robustness and effectiveness of the proposed method, whose contour optimization has higher accuracy and comprehensive value compared with that of the reference methods. This method can be used for effective post-processing to strengthen the accuracy of building extraction results.