Nonlinear dynamics in an alternating gradient guide for neutral particles