Generalized modal reduction method in dynamics of mechanical systems