Hybrid Physical-Deep Learning Model for Astronomical Inverse Problems