Can a Dove prism change the past of a single photon?