Experience sampling for building predictive user models: a comparative study

Experience sampling has been employed for decades to collect assessments of subjects' intentions, needs, and affective states. In recent years, investigators have employed automated experience sampling to collect data to build predictive user models. To date, most procedures have relied on random sampling or simple heuristics. We perform a comparative analysis of several automated strategies for guiding experience sampling, spanning a spectrum of sophistication, from a random sampling procedure to increasingly sophisticated active learning. The more sophisticated methods take a decision-theoretic approach, centering on the computation of the expected value of information of a probe, weighing the cost of the short-term disruptiveness of probes with their benefits in enhancing the long-term performance of predictive models. We test the different approaches in a field study, focused on the task of learning predictive models of the cost of interruption.

[1]  Eric Horvitz,et al.  The Lumière Project: Bayesian User Modeling for Inferring the Goals and Needs of Software Users , 1998, UAI.

[2]  Eric Horvitz,et al.  On Discarding, Caching, and Recalling Samples in Active Learning , 2007, UAI.

[3]  J. Lafferty,et al.  Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions , 2003, ICML 2003.

[4]  Ashish Kapoor,et al.  Automatic prediction of frustration , 2007, Int. J. Hum. Comput. Stud..

[5]  David J. C. MacKay,et al.  Information-Based Objective Functions for Active Data Selection , 1992, Neural Computation.

[6]  Thomas G. Dietterich,et al.  Active EM to reduce noise in activity recognition , 2007, IUI '07.

[7]  Eric Horvitz,et al.  Attention-Sensitive Alerting , 1999, UAI.

[8]  Ingrid Zukerman,et al.  Bayesian Models for Keyhole Plan Recognition in an Adventure Game , 2004, User Modeling and User-Adapted Interaction.

[9]  Claudio Gentile,et al.  Learning Probabilistic Linear-Threshold Classifiers via Selective Sampling , 2003, COLT.

[10]  James Fogarty,et al.  Examining the robustness of sensor-based statistical models of human interruptibility , 2004, CHI.

[11]  Tom Minka,et al.  A family of algorithms for approximate Bayesian inference , 2001 .

[12]  Andrew McCallum,et al.  Employing EM and Pool-Based Active Learning for Text Classification , 1998, ICML.

[13]  Mike Y. Chen,et al.  Voting with your feet : An investigative study of the relationship between place visit behavior and preference , 2006 .

[14]  Neil D. Lawrence,et al.  Fast Sparse Gaussian Process Methods: The Informative Vector Machine , 2002, NIPS.

[15]  Eric Horvitz,et al.  Learning and reasoning about interruption , 2003, ICMI '03.

[16]  William A. Gale,et al.  A sequential algorithm for training text classifiers , 1994, SIGIR '94.

[17]  Daphne Koller,et al.  Support Vector Machine Active Learning with Application sto Text Classification , 2000, ICML.

[18]  Daphne Koller,et al.  Support Vector Machine Active Learning with Applications to Text Classification , 2000, J. Mach. Learn. Res..

[19]  Ling Bao,et al.  A context-aware experience sampling tool , 2003, CHI Extended Abstracts.

[20]  Eric Horvitz,et al.  BusyBody: creating and fielding personalized models of the cost of interruption , 2004, CSCW.

[21]  Craig A. Knoblock,et al.  Active + Semi-supervised Learning = Robust Multi-View Learning , 2002, ICML.

[22]  H. Sebastian Seung,et al.  Selective Sampling Using the Query by Committee Algorithm , 1997, Machine Learning.

[23]  Eric Horvitz,et al.  Principles of mixed-initiative user interfaces , 1999, CHI '99.