Demonstration-based GUI Task Automation Through Interactive Training

Non-programming users should be able to train virtual robots to perform computer-based tasks for them. One would think that training in an all-digital noise-free environment should be easy. However, one-shot learning of a task is actually quite hard because so many mouse-clicks and key-presses are ambiguous. Also, recognizing individual actions in a task is not enough to reproduce the task, or to generalize it. For example, the intended reproduction of a copy-paste task could mean changing what is copied, or where it is pasted, or both. We propose a user-in-the-loop framework that supports vision for virtual-robots, letting a person easily "bottle" a GUI task such that it can be applied repeatedly in the future. While pure programming-by-demonstration is still unrealistic, we use quantitative and qualitative experiments to show that non-programming users are willing and effective at answering follow-up queries posed by our system. Our models of events and appearance are surprisingly simple, but are combined effectively to cope with varying amounts of supervision. The best available baseline, Sikuli Slide, struggled with the majority of the tests in our user study experiments. The prototype with our proposed approach successfully helped users accomplish simple linear tasks, complicated tasks (monitoring, looping, and mixed), and tasks that span across multiple executables. Even when both systems could ultimately perform a task, ours was trained and refined by the user in less time.