Situated Real-time Interaction with a Virtually Embodied Avatar

A well-known shortcoming of generative language models is that they can generate language which, despite being syntactically and semantically sound, is not grounded in facts [2, 17]. A growing body of recent work has shown how combining language models (LM) with external information sources makes it possible to reduce such hallucinations by letting the model directly attend to external information [15, 18, 24]; an approach commonly referred to as grounding. A common approach to grounding is to monitor a model’s output for the occurrence of certain syntactic patterns (such as the presence of agreed-upon tags) and to let an external source fill in information in the LM’s stead, after which the LM continues its generation [5, 15, 17, 19]. A similar approach may be used to let an LM-based agent interact with an external environment by monitoring for, and executing, generated actions in the environment [4,20]. These types of interactions between the LM and external systems are often enabled by LM augmentation with external plug-and-play modules, and orchestrators that coordinate LM prompts and generation [17]. This kind of external environment grounding is applica-