Towards Language-Based Modulation of Assistive Robots through Multimodal Models