Why more contact may increase cultural polarization