A multi-layered visualization language for video augmentation

There are many tasks that humans perform that involve observing video streams, as well as tracking objects or quantities related to the events depicted in the video, that can be made more transparent by the addition of appropriate drawings to a video, e.g., tracking the behavior of autonomous robots or following the motion of players across a soccer field. We describe a specification of a general means of describing groups of time-varying discrete visualizations, as well as a demonstration of overlaying those visualizations onto videos in an augmented reality manner so as to situate them in a real-world context, when such a context is available and meaningful. Creating such videos can be especially useful in the case of autonomous agents operating in the real world; we demonstrate our visualization procedures on two example robotic domains. We take the complex algorithms controlling the robots' actions in the real world and create videos that are much more informative than the original plain videos.