Aperiodicity in quantum affine $\mathfrak{gl}_n$