Loss of affineness in gels and melts