Investigation of flute instability in application to laboratory astrophysics and Z-pinch experiments