$\kappa$-deformation, affine group and spectral triples