Theory of quantum nonlinear resonance