To address the challenge of repetitive target appearance variation and frequent occlusion, existing visual tracking methods either handle corrupted samples or correct the appearance model. In this study, the authors propose a novel framework that successfully combines these two strategies. In their method, the base tracker is an improved discriminative correlation filter-based tracker, in which an independent classifier is employed to alleviate the problem of corrupted samples; the best model is selected for improvement from a group of models, which they call a ‘model colony’. The model colony is composed of models updated via different processes. The correlation output and the peak-to-sidelobe ratio are used to evaluate each model in the model colony. In addition, they propose a novel criterion called the maximum-to-others ratio for superior model selection. Experiments on 80 challenging sequences show that their tracker outperforms state-of-the-art trackers. In addition, experimental results demonstrate that their formulation significantly improves the performance of their base tracker.