Cosmological signatures of time-asymmetric gravity