Quantification of coarse-graining error in Langevin and overdamped Langevin dynamics