What Makes a Good Answer? The Role of Context in Question Answering

Question answering systems have proven to be helpful to users because they can provide succinct answers that do not require users to wade through a large number of documents. However, despite recent advances in the underlying question answering technology, the problem of designing effective interfaces has been largely unexplored. We conducted a user study to investigate this area and discovered that, overall, users prefer paragraph-sized chunks of text over just an exact phrase as the answer to their questions. Furthermore, users generally prefer answers embedded in context, regardless of the perceived reliability of the source documents. When researching a topic, increasing the amount of text returned to users significantly decreases the number of queries that they pose to the system, suggesting that users utilize supporting text to answer related questions. We believe that these results can serve to guide future developments in question answering interfaces.