Monte carlo methods for managing interactive state, action and feedback under uncertainty

Current input handling systems provide effective techniques for modeling, tracking, interpreting, and acting on user input. However, new interaction technologies violate the standard assumption that input is certain. Touch, speech recognition, gestural input, and sensors for context often produce uncertain estimates of user inputs. Current systems tend to remove uncertainty early on. However, information available in the user interface and application can help to resolve uncertainty more appropriately for the end user. This paper presents a set of techniques for tracking the state of interactive objects in the presence of uncertain inputs. These techniques use a Monte Carlo approach to maintain a probabilistically accurate description of the user interface that can be used to make informed choices about actions. Samples are used to approximate the distribution of possible inputs, possible interactor states that result from inputs, and possible actions (callbacks and feedback) interactors may execute. Because each sample is certain, the developer can specify most of the behavior of interactors in a familiar, non-probabilistic fashion. This approach retains all the advantages of maintaining information about uncertainty while minimizing the need for the developer to work in probabilistic terms. We present a working implementation of our framework and illustrate the power of these techniques within a paint program that includes three different kinds of uncertain input.