Data-driven machinery fault diagnosis methods have been successfully developed in the past decades. However, the cross-domain diagnostic problems have not been well addressed, where the training and testing data are collected under different operating conditions. Recently, domain adaptation approaches have been popularly used to bridge this gap, which extract domain-invariant features for diagnostics. Despite the effectiveness, most existing methods assume the label spaces of training and testing data are identical that indicates the fault mode sets are the same in different scenarios. In practice, new fault modes usually occur in testing, which makes the conventional methods focusing on marginal distribution alignment less effective. In order to address this problem, a deep learning-based open-set domain adaptation method is proposed in this study. Adversarial learning is introduced to extract generalized features, and an instance-level weighted mechanism is proposed to reflect the similarities of testing samples with known health states. The unknown fault mode can be effectively identified, and the known states can be also recognized. Entropy minimization scheme is further adopted to improve generalization. Experiments on two practical rotating machinery datasets validate the proposed method. The results suggest the proposed method is promising for open-set domain adaptation problems, which largely enhances the applicability of data-driven approaches in the real industries.