Crossover to non-universal microscopic spectral fluctuations in lattice gauge theory