Optical navigation for a spacecraft in a planetary system