The Green's function of the parabolic Anderson model and the continuum directed polymer