Subleading asymptotics of link spectral invariants and homeomorphism groups of surfaces