On genuine invariance learning without weight-tying