Phase Asymmetry of Andreev Spectra From Cooper-Pair Momentum