Potential well distortion and mode mixing instability in proton machines