Effect of self-magnetic fields on the nonlinear dynamics of relativistic electron beam with virtual cathode