Exploring Synergies between Visual Analytical Flow and Language Pragmatics

Interactive visual data analysis is most productive when users can focus on answering the questions they have about their data, rather than focusing on how to operate the interface to the analysis tool. One viable approach to engaging users in interactive conversations with their data is a natural language interface to visualizations. These interfaces have the potential to be both more expressive and more accessible than many other interaction paradigms. In this paper, we focus on supporting a natural flow in data conversations by considering pragmatics, or the ways in which context in a conversation influences meaning. We explore the requirements of a pragmatics component in a natural language system for visualizations and the research challenges that arise in understanding the context of data-related conversations. We then summarize how many of these challenges are generalizable to other settings and contexts involving natural language interfaces. Flow and Visual Analytics ‘Flow’ is a state of complete immersion in an activity. According to psychologist Mihàly Cskszentmihlyi, the mental state of flow is “being completely involved in an activity for its own sake. Every action, movement, and thought follows inevitably from the previous one. Your whole being is involved, and you’re using your skills to the utmost (Csikszentmihalyi 1991).” A person in a state of flow is completely absorbed in their activity, to the exclusion of everything else. Activities in which people can use their creative abilities are especially likely to lead to a state of flow. However, with the world inundated with mobile devices, social media and various forms of multi-tasking, much of our lives consist of endless interruptions, hindering productivity and reducing our ability to simply enjoy the moment. These interruptions have ramifications while using computer software, with the product or interface itself often getting in the way. In data analytics, these interruptions can often hinder a user from having a fluid conversation with her data and exploring answers to questions she may have along the way. The excitement and flow of finding insights in the data are often disrupted by endless wizards, dialog boxes or long wait times. Instead of engaging with her data to answer quesCopyright c © 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. tions, the user spends her time thinking about how to operate elements of the interface. Visual analytics tools help to engage a user in the flow of analysis. The human visual system is extremely adept at processing visual information such as color, shape, and size. By encoding data in these visual attributes, visualization tools offload cognitive work to the perceptual system, enabling users to focus on answering questions about their data rather than reading and comparing data values. A critical component of any visualization tool is interactivity. Rarely can a user’s complex questions be answered by a single static chart. Most of the time, a user will need to interactively change the data display by filtering, navigating, and seeking details-on-demand, to focus on a small portion of the data relevant to the question at hand. Moreover, a user may create and explore a whole series of charts to answer new questions that arise. It is during these interactions that it is critical to keep users in the flow of conversation. Classic interaction techniques such as dynamic queries (Ahlberg, Williamson, and Shneiderman 1992) were designed precisely to keep the user’s focus on the data display rather than on external interface widgets. Nonetheless, interacting with powerful analytic tools can be challenging and often requires substantial user practice to become proficient. A critical requirement to facilitate analytical flow is for the system to answer a user’s question intelligently without expecting the user to be a skilled statistician or database expert. It has long been known that inexperienced users have difficulty using native database query languages such as SQL to express their data needs (Li and Jagadish 2014). But, even with visual drag-and-drop interfaces, users can still struggle to express their data-oriented questions in terms of tool operations (Grammel, Tory, and Storey 2010). This can occur for a variety of reasons; for instance, the question may be vague rather than clearly formulated, the entry point for the question may not match that required by the tool (e.g., thinking about the data attributes involved when the tool requires you to first choose a chart type), there may be a mismatch between the terminology in the question versus naming of the tool’s functions, or the user may simply not know what set of operations is needed to answer the question. Our goal is to build intelligent analytical tools without barriers that get in the way of people asking and answering questions. In this way, we hope to help people do what they do best: think and create. Natural Language Interfaces Natural language interfaces to data have emerged as a promising new way of interacting with data and performing analytics. These interfaces take, as input, an utterance formulated in natural language and return an appropriate answer. This approach is promising in maintaining flow, as users may be able to express their questions more easily in natural language rather than translating those questions to appropriate system commands. There are several companies actively investing in this space. IBM has released Watson Analytics that features a natural language interface for starting an analysis. Microsoft released Q&A in Power BI to allow users to type in natural language queries of their data such as “sales per sq ft by store in NC”. ThoughtSpot provides a natural language search engine for data. Narrative Science has developed a product to generate natural language summaries of visualizations. Each of these systems is interesting but has fundamental limitations. Most return a minimally interactive visualization in response to queries, meaning the answer needs to be exactly right rather than approximate. Many require experts to perform modeling before the systems are effective. None are richly integrated with a self-service analysis tool in a manner that allows natural language interactions to become part of a richer visual cycle of analysis. Creating robust natural language interfaces are often difficult to realize, as they have to handle difficult problems inherent in the task of automatically interpreting natural language. In addition, natural language expressions are often diverse and imprecise, requiring extensive knowledge and sophisticated reasoning for computers to interpret them. Another challenge is handling the disconnect between the user’s model and how the system interprets the user’s intent. One of the most important aspects of maintaining flow, is how the natural language interface handles this ambiguity and allows for the user to correct the system if the interpretation is wrong. One way to deal with ambiguity is to make a ‘best guess’ at the user’s intent so that a chart can be shown right away. The Articulate system (Sun et al. 2010) accomplished this by extracting syntactic and semantic information from a user’s query, applying a supervised learning algorithm to translate that into an understanding of their intention, and then generating an appropriate visualization. However, Articulate focused primarily on generating a visualization; it enabled very little interaction with the visualization and therefore fell short of supporting cycles of conversation with one’s data. DataTone (Gao et al. 2015) improved the analysis flow by making a best guess at the user’s intent, producing a chart according to that best guess, and then providing ambiguity widgets through which the user could change their chart if the system’s guess was incorrect. More recently, another system called Eviza (Setlur et al. 2016) provided a natural language interface for interacting with an existing visualization rather than starting from a blank sheet and simply asking questions of an entire data set. This reduces the scope of the problem and makes a more useful answer more likely. The system also built in rich domain awareness of time, space, and quantitative reasoning as well as linking into existing knowledge bases like Wolfram. This reduces the need for end users to do sophisticated modeling prior to using the system while supporting more expressive queries. However, there are other aspects of natural language interfaces that could further help with flow. Conversation frequently consists of a series of related utterances, often referring to past references and context (Allen 1982). As part of this conversational flow, the semantics of a particular query can be influenced by each other, a concept known as language pragmatics.