Nowadays, authentication systems are usually required to provide continuous, contactless, and non-intrusive services. In this paper, we propose RF-Badge, a vital sign-based authentication scheme on human subjects to meet the above requirements by using RFID technology. We consider two biometric features with individual diversity to characterize the vital sign of users, including the movement effect from respiration and the reflection effect from organs, especially the heart. To derive the movement effect from respiration, we build a phase-based geometric model to restore the fine-grained badge moving trace as the feature. To derive the reflection effect from human internal organs, we extract the reflection signal from the original signal and generate the spectrum as the feature. Besides, to deal with the feature deviation in different physical conditions of users, we propose a multi-condition network (MCNet) to further guarantee the generalization of RF-Badge. We implement a prototype system and evaluate the performance in real environments. The experiment results show that our system achieves the average false positive rate (FPR) of 3.9% and false negative rate (FNR) of 3.3% for continuous authentication within four signal cycles.