ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets