Using Interaction to Improve Intelligence : How Intelligent Systems Should Ask Users for Input

Intelligent systems will often need to collect input from users, to provide labels for training data or to correct mistakes the system makes. One interesting avenue of research is how to formulate the questions an intelligent system asks a user, in order to obtain the most accurate responses. In this paper, we study the impact of varying 5 dimensions of questions on response accuracy: indicating uncertainty, amount of context, level of context, suggesting an answer and asking for supplemental information. In a study of an email sorting task, we show that there is a combination that results in higher levels of accuracy than other combinations and validate this combination in a comparison to questions that a panel of HCI and email experts chose. The contributions of the paper are the approach to determine the best combination of dimensions, the validated combination, and a demonstration of how this type of question interaction can improve intelligent systems.

[1]  Bernt Schiele,et al.  Evaluating the Effects of Displaying Uncertainty in Context-Aware Applications , 2004, UbiComp.

[2]  Mica R. Endsley,et al.  Being Certain About Uncertainty: How the Representation of System Reliability Affects Pilot Decision Making , 1998 .

[3]  James Fogarty,et al.  Amplifying community content creation with mixed initiative information extraction , 2009, CHI.

[4]  Weng-Keen Wong,et al.  Integrating rich user feedback into intelligent user interfaces , 2008, IUI '08.

[5]  M. Couper,et al.  METHODS FOR TESTING AND EVALUATING SURVEY QUESTIONS , 2004 .

[6]  L. Steels.  Evolving grounded communication for robots , 2003, Trends in Cognitive Sciences.

[7]  Takayuki Kanda,et al.  A semi-autonomous communication robot — A field trial at a train station , 2008, 2008 3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[8]  Robin Jeffries,et al.  User interface evaluation in the real world: a comparison of four techniques , 1991, CHI.

[9]  Thomas G. Dietterich,et al.  Toward harnessing user feedback for machine learning , 2007, IUI '07.

[10]  Thomas G. Dietterich,et al.  Predicting User Tasks : I Know What You ’ re Doing ! , 2005 .

[11]  Gerard Salton,et al.  Improving retrieval performance by relevance feedback , 1997, J. Am. Soc. Inf. Sci..

[12]  Henrik I. Christensen,et al.  Bringing Together Human and Robotic Environment Representations - A Pilot Study , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[13]  Paul R. Cohen,et al.  The RADAR Test Methodology: Evaluating a Multi-Task Machine Learning System with Humans in the Loop , 2006 .

[14]  Krzysztof Z. Gajos,et al.  Preference elicitation for interface optimization , 2005, UIST.

[15]  Mark C. Torrance,et al.  Natural communication with robots , 1994 .