Nonlinear instability dynamics in a high-density, high-beta plasma