Restricted likelihood ratio testing in linear mixed models with general error covariance structure