When Visual Attention Hurts