Nonlinear analysis of microscopic instabilities in fiber-reinforced composite materials