Simulation and analysis of the flower-like instability in the single-feedback mirror experiment with rubidium vapor