On the nonexistence of semi-regular relative difference sets