If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills

Interview chatbots engage users in a text-based conversation to draw out their views and opinions. It is, however, challenging to build effective interview chatbots that can handle user free-text responses to open-ended questions and deliver engaging user experience. As the first step, we are investigating the feasibility and effectiveness of using publicly available, practical AI technologies to build effective interview chatbots. To demonstrate feasibility, we built a prototype scoped to enable interview chatbots with a subset of active listening skills-the abilities to comprehend a user's input and respond properly. To evaluate the effectiveness of our prototype, we compared the performance of interview chatbots with or without active listening skills on four common interview topics in a live evaluation with 206 users. Our work presents practical design implications for building effective interview chatbots, hybrid chatbot platforms, and empathetic chatbots beyond interview tasks.

[1]  Joseph Weizenbaum,et al.  ELIZA—a computer program for the study of natural language communication between man and machine , 1966, CACM.

[2]  Eric P. Xing,et al.  Harnessing Deep Neural Networks with Logic Rules , 2016, ACL.

[3]  Kallirroi Georgila,et al.  SimSensei kiosk: a virtual human interviewer for healthcare decision support , 2014, AAMAS.

[4]  Louis-Philippe Morency,et al.  It's only a computer: Virtual humans increase willingness to disclose , 2014, Comput. Hum. Behav..

[5]  Denny Britz,et al.  Generating Long and Diverse Responses with Neural Conversation Models , 2017, ArXiv.

[6]  Gloria Mark,et al.  Trusting Virtual Agents , 2019, ACM Trans. Interact. Intell. Syst..

[7]  Xiaoyan Zhu,et al.  Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory , 2017, AAAI.

[8]  Dragomir R. Radev,et al.  LexRank: Graph-based Lexical Centrality as Salience in Text Summarization , 2004, J. Artif. Intell. Res..

[9]  Gloria Mark,et al.  Tell Me About Yourself , 2019, ACM Trans. Comput. Hum. Interact..

[10]  Jaime Teevan,et al.  Supporting Workplace Detachment and Reattachment with Conversational Intelligence , 2018, CHI.

[11]  Joelle Pineau,et al.  Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models , 2015, AAAI.

[12]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[13]  Joseph Weizenbaum,et al.  and Machine , 1977 .

[14]  S. Shyam Sundar,et al.  Machine Heuristic: When We Trust Computers More than Humans with Our Personal Information , 2019, CHI.

[15]  Ankur Taly,et al.  Axiomatic Attribution for Deep Networks , 2017, ICML.

[16]  GratchJonathan,et al.  It's only a computer , 2014 .

[17]  Timothy Bickmore,et al.  An Internet-Based Virtual Coach to Promote Physical Activity Adherence in Overweight Adults: Randomized Controlled Trial , 2012, Journal of medical Internet research.

[18]  Iadis International.  INTRODUCING “ ACTIVE LISTENING ” TO INSTANT MESSAGING AND E-MAIL : BENEFITS AND LIMITATIONS , 2012 .

[19]  E. Weigand.  The Routledge Handbook of Language and Dialogue , 2017 .

[20]  Greg Ver Steeg,et al.  Anchored Correlation Explanation: Topic Modeling with Minimal Domain Knowledge , 2016, TACL.

[21]  Ina Fourie.  Health Informatics: A Patient‐centered Approach to Diabetes , 2011 .

[22]  Xinya Du,et al.  Learning to Ask: Neural Question Generation for Reading Comprehension , 2017, ACL.

[23]  Petr Sojka,et al.  Gensim -- Statistical Semantics in Python , 2011 .

[24]  Bert Decker.  How to communicate effectively , 1989 .

[25]  Carl R. Rogers,et al.  Active Listening , 2015 .

[26]  Michael C. McCord,et al.  Deep parsing in Watson , 2012, IBM J. Res. Dev..

[27]  Oliver Lemon,et al.  Hybrid chat and task dialogue for more engaging HRI using reinforcement learning , 2017, 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).

[28]  Clifford Nass,et al.  Computers are social actors , 1994, CHI '94.

[29]  Harry Weger,et al.  Active Listening in Peer Interviews: The Influence of Message Paraphrasing on Perceptions of Listening Skill , 2010 .

[30]  Nan Hua,et al.  Universal Sentence Encoder , 2018, ArXiv.

[31]  D. S. Jones,et al.  Elementary information theory , 1979 .

[32]  Wei-Ying Ma,et al.  Topic Aware Neural Response Generation , 2016, AAAI.

[33]  Jay F. Nunamaker,et al.  Embodied Conversational Agent-Based Kiosk for Automated Interviewing , 2011, J. Manag. Inf. Syst..

[34]  Chris Speed,et al.  The Ethnobot: Gathering Ethnographies in the Age of IoT , 2018, CHI.

[35]  Deborah I Thompson,et al.  Use of Relational Agents to Improve Family Communication in Type 1 Diabetes: Methods , 2016, JMIR research protocols.

[36]  Joonhwan Lee,et al.  Comparing Data from Chatbot and Web Surveys: Effects of Platform and Conversational Style on Survey Response Quality , 2019, CHI.

[37]  Minlie Huang,et al.  Learning to Ask Questions in Open-domain Conversational Systems with Typed Decoders , 2018, ACL.

[38]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[39]  Yukari Yamakawa,et al.  Generating Questions and Multiple-Choice Answers using Semantic Analysis of Texts , 2016, COLING.

[40]  Joelle Pineau,et al.  Training End-to-End Dialogue Systems with the Ubuntu Dialogue Corpus , 2017, Dialogue Discourse.

[41]  Jiebo Luo,et al.  Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Media , 2018, CHI.

[42]  Shimei Pan,et al.  TIARA: Interactive, Topic-Based Visual Text Summarization and Analysis , 2012, TIST.

[43]  Maxine Eskénazi,et al.  Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders , 2017, ACL.

[44]  Gloria Mark,et al.  Confiding in and Listening to Virtual Agents: The Effect of Personality , 2017, IUI.

[45]  Wijnand A. IJsselsteijn,et al.  Caring for Vincent: A Chatbot for Self-Compassion , 2019, CHI.

[46]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[47]  Geoffrey Zweig,et al.  Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning , 2017, ACL.

[48]  J. P. Morgan,et al.  Design and Analysis: A Researcher's Handbook , 2005, Technometrics.

[49]  Traum. David,et al.  Computational Approaches to Dialogue , 2017 .

[50]  Jonathan Grudin,et al.  Chatbots, Humbots, and the Quest for Artificial General Intelligence , 2019, CHI.

[51]  Krithika Jagannath,et al.  Artificially Intelligent (AI) Tutors in the Classroom: A Need Assessment Study of Designing Chatbots to Support Student Learning , 2019, PACIS.

[52]  Stefan Dietze,et al.  Understanding Malicious Behavior in Crowdsourcing Platforms: The Case of Online Surveys , 2015, CHI.

[53]  William Aspray,et al.  Relational Agents for Chronic Disease Self-Management , 2010 .

[54]  Guoyin Wang,et al.  Joint Embedding of Words and Labels for Text Classification , 2018, ACL.

[55]  Jason Weston,et al.  Learning End-to-End Goal-Oriented Dialog , 2016, ICLR.

[56]  K. Figl,et al.  “Active listening” in written online communication - a case study in a course on “soft skills for computer scientists” , 2008, 2008 38th Annual Frontiers in Education Conference.

[57]  Shimei Pan,et al.  Topic and keyword re-ranking for LDA-based topic modeling , 2009, CIKM.

[58]  Evgeniy Gabrilovich,et al.  A Review of Relational Machine Learning for Knowledge Graphs , 2015, Proceedings of the IEEE.

[59]  Shwetak N. Patel,et al.  FarmChat , 2018, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[60]  Alexander I. Rudnicky,et al.  Learning Conversational Systems that Interleave Task and Non-Task Content , 2017, IJCAI.

[61]  Timothy R. Hill,et al.  Intelligent Career Advisers in Your Pocket? A Need Assessment Study of Chatbots for Student Career Advising , 2019, AMCIS.

[62]  Zhendong Mao,et al.  Knowledge Graph Embedding: A Survey of Approaches and Applications , 2017, IEEE Transactions on Knowledge and Data Engineering.

[63]  Michelle X. Zhou,et al.  Who should be my teammates: using a conversational agent to understand individuals and help teaming , 2019, IUI.

[64]  Patrick Gebhard,et al.  Exploring interaction strategies for virtual characters to induce stress in simulated job interviews , 2014, AAMAS.

[65]  Christine Bauer,et al.  Introducing "Active Listening" to Instant Messaging and E-Mail: Benefits and Limitations , 2010 .