Learning Less Generalizable Patterns for Better Test-Time Adaptation