Towards a Global Springer Theory II: the double affine action