The refined Swampland Distance Conjecture in Calabi-Yau moduli spaces