Nonlinear dynamics of flute modes and self-organization phenomena in turbulent magnetized plasma