The asymmetric quantum Rabi model in the polaron picture