Refined bounds on MeV-scale thermal dark sectors from BBN and the CMB