Matrix-free large-scale Bayesian inference in cosmology