Conversations with Documents: An Exploration of Document-Centered Assistance

The role of conversational assistants has become more prevalent in helping people increase their productivity. Document-centered assistance, for example to help an individual quickly review a document, has seen less significant progress, even though it has the potential to tremendously increase a user's productivity. This type of document-centered assistance is the focus of this paper. Our contributions are three-fold: (1) We first present a survey to understand the space of document-centered assistance and the capabilities people expect in this scenario. (2) We investigate the types of queries that users will pose while seeking assistance with documents, and show that document-centered questions form the majority of these queries. (3) We present a set of initial machine learned models that show that (a) we can accurately detect document-centered questions, and (b) we can build reasonably accurate models for answering such questions. These positive results are encouraging, and suggest that even greater results may be attained with continued study of this interesting and novel problem space. Our findings have implications for the design of intelligent systems to support task completion via natural interactions with documents.

[1]  Dongyan Zhao,et al.  Coupled Context Modeling for Deep Chit-Chat: Towards Conversations between Human and Computer , 2018, KDD.

[2]  Susan T. Dumais,et al.  Automatic Identification and Contextual Reformulation of Implicit System-Related Queries , 2016, SIGIR.

[3]  Susan T. Dumais,et al.  Characterizing Search Behavior in Productivity Software , 2018, CHIIR.

[4]  Xiaofei He,et al.  Query rewriting using active learning for sponsored search , 2007, SIGIR.

[5]  Jimmy J. Lin,et al.  Data Augmentation for BERT Fine-Tuning in Open-Domain Question Answering , 2019, ArXiv.

[6]  Fabrizio Silvestri,et al.  Context- and Content-aware Embeddings for Query Rewriting in Sponsored Search , 2015, SIGIR.

[7]  Ming-Wei Chang,et al.  Natural Questions: A Benchmark for Question Answering Research , 2019, TACL.

[8]  M. de Rijke,et al.  Learning to Transform, Combine, and Reason in Open-Domain Question Answering , 2019, BNAIC/BENELEARN.

[9]  Percy Liang,et al.  Know What You Don’t Know: Unanswerable Questions for SQuAD , 2018, ACL.

[10]  Stefan Feuerriegel,et al.  RankQA: Neural Question Answering with Answer Re-Ranking , 2019, ACL.

[11]  Jimmy J. Lin,et al.  End-to-End Open-Domain Question Answering with BERTserini , 2019, NAACL.

[12]  ZaragozaHugo,et al.  The Probabilistic Relevance Framework , 2009 .

[13]  Paul Green,et al.  Development and Evaluation of Automotive Speech Interfaces: Useful Information from the Human Factors and the Related Literature , 2013 .

[14]  Chinnadhurai Sankar.  Modeling Non-Goal Oriented Dialog With Discrete Attributes , 2018 .

[15]  Ryen W. White,et al.  Mercury: Empowering Programmers' Mobile Work Practices with Microproductivity , 2019, UIST.

[16]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.

[17]  Philip Bachman,et al.  NewsQA: A Machine Comprehension Dataset , 2016, Rep4NLP@ACL.

[18]  Tero Jokela,et al.  A Diary Study on Combining Multiple Information Devices in Everyday Activities and Tasks , 2015, CHI.

[19]  Yoshua Bengio,et al.  HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering , 2018, EMNLP.

[20]  Kyunghyun Cho,et al.  SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine , 2017, ArXiv.

[21]  Susan T. Dumais,et al.  Characterizing Email Search using Large-scale Behavioral Logs and Surveys , 2017, WWW.

[22]  Ludovic Denoyer,et al.  Unsupervised Question Answering by Cloze Translation , 2019, ACL.

[23]  Eunsol Choi,et al.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension , 2017, ACL.

[24]  Jason Weston,et al.  Reading Wikipedia to Answer Open-Domain Questions , 2017, ACL.

[25]  Danqi Chen,et al.  CoQA: A Conversational Question Answering Challenge , 2018, TACL.

[26]  Willie Brink,et al.  Towards Automating Healthcare Question Answering in a Noisy Multilingual Low-Resource Setting , 2019, ACL.

[27]  I. V. Ramakrishnan,et al.  Capti-speak: a speech-enabled web screen reader , 2015, W4A.

[28]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[29]  Jason Weston,et al.  Large-scale Simple Question Answering with Memory Networks , 2015, ArXiv.

[30]  Yi Yang,et al.  WikiQA: A Challenge Dataset for Open-Domain Question Answering , 2015, EMNLP.

[31]  R'emi Louf,et al.  HuggingFace's Transformers: State-of-the-art Natural Language Processing , 2019, ArXiv.

[32]  Hwee Tou Ng,et al.  Improving the Robustness of Question Answering Systems to Question Paraphrasing , 2019, ACL.

[33]  Jianfeng Gao,et al.  A Human Generated MAchine Reading COmprehension Dataset , 2018 .

[34]  Chris Dyer,et al.  The NarrativeQA Reading Comprehension Challenge , 2017, TACL.

[35]  Jaime Teevan,et al.  An Exploration of Speech-Based Productivity Support in the Car , 2019, CHI.

[36]  John C. Tang,et al.  Mobile taskflow in context: a screenshot study of smartphone usage , 2010, CHI.

[37]  Hugo Zaragoza,et al.  The Probabilistic Relevance Framework: BM25 and Beyond , 2009, Found. Trends Inf. Retr..

[38]  Yuan Li,et al.  Probabilistic models for aggregating crowdsourced annotations , 2019 .

[39]  Jian Zhang,et al.  SQuAD: 100,000+ Questions for Machine Comprehension of Text , 2016, EMNLP.

[40]  Linda Di Geronimo,et al.  Surveying personal device ecosystems with cross-device applications in mind , 2016, PerDis.

[41]  Ryen W. White,et al.  VERSE: Bridging Screen Readers and Voice Assistants for Enhanced Eyes-Free Web Search , 2019, ASSETS.