A quantum spin liquid with a $Z_2$ topological order has long been thought to be important for the application of quantum computing and may be related to high-temperature superconductivity. While a two-dimensional kagome antiferromagnet may host such a state, strong experimental evidences are still lacking. Here we show that the spin excitations from the kagome planes in magnetically ordered Cu$_4$(OD)$_6$FBr and non-magnetically ordered Cu$_3$Zn(OD)$_6$FBr are similarly gapped although the content of inter-kagome-layer Cu$^{2+}$ ions changes dramatically. This suggests that the spin triplet gap and continuum of the intrinsic kagome antiferromagnet are robust against the interlayer magnetic impurities. Our results show that the ground state of Cu$_3$Zn(OD)$_6$FBr is a gapped quantum spin liquid with Z$_2$ topological order.