Model-independent cosmological constraints from growth and expansion