Nonaffinity controls critical slowing down and rheology near the onset of rigidity