Activity recognition in desktop environments

Knowledge workers are struggling in the information flood. There is a growing interest in intelligent desktop environments that help knowledge workers organize their daily life. Intelligent desktop environments allow the desktop user to define a set of “activities” that characterize the user’s desktop work. These environments then attempt to identify the current activity of the user in order to provide various kinds of assistance. TaskTracer [Dragunov et al., 2005] is one of these efforts. We classify activities into three levels: task as in TaskTracer, workflow, and operation. In this thesis, we concentrate on the two higher levels of activity – “task” and “workflow” – and we present an activity recognition solution for each of them. To recognize tasks, we employ various supervised learning algorithms. The first approach is based on batch training, which has a heavy requirement for CPU time and memory. It is also insensitive to user feedback, and it is likely to make repeated mistakes. To address these issues, we propose a novel online learning algorithm that is able to incorporate a richer set of features than our previous predictors. We prove an error bound for the algorithm and present experimental results that show improved accuracy and a 180-fold speedup on real user data. To recognize workflows, we first mine frequent workflow models from the TaskTracer event log. Discovering desktop workflows is difficult because they unfold over extended periods of time (days or weeks) and they are interleaved with many other workflows because of user multi-tasking. We describe an approach to discovering desktop workflows based on rich instrumentation of information flow actions such as copy/paste, SaveAs, file copy, attach file to email message, and save attachment. These actions allow us to construct a graph whose nodes are files, email messages, and web pages and whose edges are these information flow actions. A class of workflows that we call work procedures can be discovered by applying graph mining algorithms to find frequent subgraphs. After finding the workflow models, we create a Logical Hidden Markov Model (LHMM) for each mined model to represent the workflow in a form suitable for real-time recognition using the WARP relational inference algorithm [Natarajan et al., 2008].