We present a computer vision tool to improve the clinical outcome of patients undergoing radiation therapy for prostate cancer by improving irradiation technique. While intensity modulated radiotherapy (IMRT) allows one to irradiate a specific region in the body with high accuracy, it is still difficult to know exactly where to aim the radiation beam on every day of the 30~40 treatments that are necessary. This paper presents a geometric model-based technique to accurately segment the prostate and other surrounding structures in a daily serial CT image, compensating for daily motion and shape variation. We first acquire a collection of serial CT scans of patients undergoing external beam radiotherapy, and manual segmentation of the prostate and other nearby structures by radiation oncologists. Then we train shape and local appearance models for the structures of interest. When new images are available, an iterative algorithm is applied to locate the prostate and surrounding structures automatically. Our experimental results show that excellent matches can be given to the prostate and surrounding structure. Convergence is declared after 10 iterations. For 256 x 256 images, the mean distance between the hand-segmented contour and the automatically estimated contour is about 1.5 pixels (2.44 mm), with variance about 0.6 pixel (1.24 mm).