Diggle (1988) described how the empirical semi-variogram of ordinary least squares residuals can be used to suggest an appropriate serial correlation structure in stationary linear mixed models. In this paper, this approach is extended to non-stationary models which include random effects other than intercepts, and will be applied to prostate cancer data, taken from the Baltimore Longitudinal Study of Aging. A simulation study demonstrates the effectiveness of this extended variogram for improving the covariance structure of the linear mixed model used to describe the prostate data.