Scanning tunneling microscopy visualizations of quasiparticle interference (QPI) enable powerful insights into the k-space properties of superconducting, topological, Rashba and other exotic electronic phases, but their reliance on impurities acting as scattering centers is rarely scrutinized. Here we investigate QPI at the vacuum-cleaved (001) surface of the Dirac semimetal ZrSiS. We find that interference patterns around impurities located on the Zr and S lattice sites appear very different, and can be ascribed to selective scattering of different sub-sets of the predominantly Zr 4d-derived band structure, namely the m = 0 and m = +/-1 components. We show that the selectivity of scattering channels requires an explanation beyond the different bands' orbital characteristics and their respective charge density distributions over Zr and S lattices sites. Importantly, this result shows that the usual assumption of generic scattering centers allowing observations of quasiparticle interference to shed light indiscriminately and isotropically upon the \textit{q}-space of scattering events does not hold, and that the scope and interpretation of QPI observations can therefore be be strongly contingent on the material defect chemistry. This finding promises to spur new investigations into the quasiparticle scattering process itself, to inform future interpretations of quasiparticle interference observations, and ultimately to aid the understanding and engineering of quantum electronic transport properties.