Estimating negative variance components from Gaussian and non-Gaussian data: A mixed models approach

The occurrence of negative variance components is a reasonably well understood phenomenon in the case of linear models for hierarchical data, such as variance-component models in designed experiments or linear mixed models for longitudinal data. In many cases, such negative variance components can be translated as negative within-unit correlations. It is shown that negative variance components, with corresponding negative associations, can occur in hierarchical models for non-Gaussian outcomes as well, such as repeated binary data or counts. While this feature poses no problem for marginal models, in which the mean and correlation functions are modeled directly and separately, the issue is more complicated in, for example, generalized linear mixed models. This owes in part to the non-linear nature of the link function, non-constant residual variance stemming from the mean-variance link, and the resulting lack of closed-form expressions for the marginal correlations. It is established that such negative variance components in generalized linear mixed models can occur in practice and that they can be estimated using standard statistical software. Marginal-correlation functions are derived. Important implications for interpretation and model choice are discussed. Simulations and the analysis of data from a developmental toxicity experiment underscore these results.