Asking Questions and Developing Trust

In a number of domains, researchers instrument an interface or the environment with software and hardware sensors to collect observational data, but it is often quite hard to label that data accurately. We are interested in how an agent can ask many humans at once for help to label its data and we present two studies towards this goal. First, we investigate how a computer can automatically elicit labels from users as they interact with different technologies. Then, we present a study comparing different algorithms for how an agent decides which users to trust when a lot of people answer the agent’s questions and the answers are conflicting. We discuss the implications of the results in each of these studies and present some ideas for future work towards agents asking humans questions.

[1]  Thomas G. Dietterich,et al.  Predicting User Tasks : I Know What You ’ re Doing ! , 2005 .

[2]  David A. Cohn,et al.  Improving generalization with active learning , 1994, Machine Learning.

[3]  Manfred K. Warmuth,et al.  The weighted majority algorithm , 1989, 30th Annual Symposium on Foundations of Computer Science.

[4]  Mica R. Endsley,et al.  Being Certain About Uncertainty: How the Representation of System Reliability Affects Pilot Decision Making , 1998 .

[5]  Bernt Schiele,et al.  Evaluating the Effects of Displaying Uncertainty in Context-Aware Applications , 2004, UbiComp.

[6]  Jure Leskovec,et al.  The dynamics of viral marketing , 2005, EC '06.

[7]  Dragos D. Margineantu,et al.  Active Cost-Sensitive Learning , 2005, IJCAI.

[8]  Thomas G. Dietterich,et al.  Toward harnessing user feedback for machine learning , 2007, IUI '07.

[9]  Gerard Salton,et al.  Improving retrieval performance by relevance feedback , 1997, J. Am. Soc. Inf. Sci..

[10]  Thomas S. Huang,et al.  Relevance feedback: a power tool for interactive content-based image retrieval , 1998, IEEE Trans. Circuits Syst. Video Technol..

[11]  Sean M. McNee,et al.  Confidence Displays and Training in Recommender Systems , 2003, INTERACT.

[12]  Manuela M. Veloso,et al.  Interactive robot task training through dialog and demonstration , 2007, 2007 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[13]  Brad A. Myers,et al.  Successful User Interfaces for RADAR , 2008 .

[14]  Thomas G. Dietterich What is machine learning? , 2020, Archives of Disease in Childhood.

[15]  Paul R. Cohen,et al.  The RADAR Test Methodology: Evaluating a Multi-Task Machine Learning System with Humans in the Loop , 2006 .

[16]  Charles E. Thorpe,et al.  Collaborative control: a robot-centric model for vehicle teleoperation , 2001 .

[17]  Chrystopher L. Nehaniv,et al.  Teaching robots by moulding behavior and scaffolding the environment , 2006, HRI '06.

[18]  Jaime G. Carbonell,et al.  Proactive learning: cost-sensitive active learning with multiple imperfect oracles , 2008, CIKM '08.

[19]  Nicolò Cesa-Bianchi,et al.  Gambling in a rigged casino: The adversarial multi-armed bandit problem , 1995, Proceedings of IEEE 36th Annual Foundations of Computer Science.