When and why vision-language models behave like bags-of-words, and what to do about it?