W HEN AND WHY V ISION -L ANGUAGE M ODELS BE HAVE LIKE B AGS - OF -W ORDS , AND WHAT TO DO ABOUT IT ?