Can broken rotational invariance be reconciled with inflation?