The complex structure of gas, metals, and dust in the interstellar and circumgalactic medium (ISM and CGM, respectively) in star-forming galaxies can be probed by Lyα emission and absorption, low-ionization interstellar (LIS) metal absorption, and dust reddening E(B − V). We present a statistical analysis of the mutual correlations among Lyα equivalent width (EWLyα ), LIS equivalent width (EWLIS), and E(B − V) in a sample of 157 star-forming galaxies at z ∼ 2.3. With measurements obtained from individual deep rest-UV spectra and spectral energy distribution modeling, we find that the tightest correlation exists between EWLIS and E(B − V), although correlations among all three parameters are statistically significant. These results signal a direct connection between dust and metal-enriched H i gas and that they are likely cospatial. By comparing our results with the predictions of different ISM/CGM models, we favor a dusty ISM/CGM model where dust resides in H i gas clumps and Lyα photons escape through the low H i covering fraction/column density intraclump medium. Finally, we investigate the factors that potentially contribute to the intrinsic scatter in the correlations studied in this work, including metallicity, outflow kinematics, Lyα production efficiency, and slit loss. Specifically, we find evidence that scatter in the relationship between EWLyα and E(B − V) reflects the variation in the metal–to–H i covering fraction ratio as a function of metallicity and the effects of outflows on the porosity of the ISM/CGM. Future simulations incorporating star formation feedback and the radiative transfer of Lyα photons will provide key constraints on the spatial distributions of neutral hydrogen gas and dust in the ISM/CGM structure.