In the heavily hole-doped iron-based superconductors $A$Fe$_2$As$_2$ ($A=$ K, Rb, and Cs), the electron effective mass increases rapidly with alkali-ion radius. To study how the mass enhancement affects the superconducting state, we measure the London penetration depth $\lambda(T)$ in clean crystals of $A$Fe$_2$As$_2$ down to low temperature $T\sim0.1$ K. In all systems, the superfluid stiffness $\rho_s(T)=\lambda^2(0)/\lambda^2(T)$ can be approximated by a power-law $T$ dependence at low temperatures, indicating the robustness of strong momentum anisotropy in the superconducting gap $\Delta(k)$. The power $\alpha$ increases from $\sim1$ with mass enhancement and approaches an unconventional exponent $\alpha\sim 1.5$ in the heaviest CsFe$_2$As$_2$. This appears to be a hallmark of superconductors near antiferromagnetic quantum critical points, where the quasiparticles excited across the anisotropic $\Delta(k)$ are significantly influenced by the momentum dependence of quantum critical fluctuations.