Most of the state-of-the-art visual simultaneous localization and mapping (V-SLAM) algorithms perform well in structured environments. However, the assumption of a static world limits further practical application. In this paper, we present RS-SLAM, a robust semantic RGB-D SLAM system. RS-SLAM is able to detect both moving and movable objects for high accuracy localization. A semantic segmentation model is applied to recognize the dynamic objects. Specifically, the segmentation result is refined using the context information based on Bayesian update, which provides the more accurate extraction of the interested region. With the refined results, a kind of movable object detection approach is developed. In addition, an updating strategy during mapping is introduced and a 3D semantic OctoMap only with a static world is built in real-time. Our SLAM system is evaluated both on public datasets and real-world environments. The results show that RS-SLAM outperforms the accuracy of standard visual SLAM baselines and is capable of constructing a clean static semantic OctoMap in a dynamic environment.