Structured dispersion and validity in linear inference