Adjoint-aided inference of Gaussian process driven differential equations