Renormalization group method in the theory of dynamical systems