Non-Equilibrium Learning with Heterogeneous Priors