Dynamical phase transition in Floquet optical bistable systems: An approach from finite-size quantum systems