Origin of Non-Reciprocal Response in Fiber Optic Parametric Amplifiers