Referring To World Objects With Text And Pictures

It often makes sense to employ both text and pictures when referring to world objects. In this paper, we present a model for referring which is based on the assumption that concepts may be activated not only by text, but also by pictures and text-picture combinations. By means of a case study, we demonstrate that failure and success of referring acts can be explained by the user's ability to infer certain links between mental representations and object descriptions. Finally, we show how the model has been incorporated into a plan-based multimedia presentation system by defining operators for concept activation.