This paper proposes a robust method for tracking an object contour in a sequence of images. In this method, both object extraction and tracking problems can be solved simultaneously. Furthermore, it is applicable to the tracking of arbitrary shapes since it does not need a priori knowledge about the object shapes. In the contour tracking, energy-minimizing elastic contour models are utilized, which is newly presented in this paper. The contour tracking is formulated as an optimization problem to find the position that minimizes both the elastic energy of its model and the potential energy derived from the edge potential image that includes a target object contour. We also present an algorithm which efficiently solves energy minimization problems within a dynamic programming framework. The algorithm enables us to obtain optimal solution even when the variables to be optimized are not ordered. We show the validity and usefulness of the proposed method with some experimental results.