Sharp adaptation for spherical inverse problems with applications to medical imaging