Intelligent Assistant for Exploring Data Visualizations

Visualization, while an effective tool for identifying patterns and insights, requires expert knowledge due to challenges faced when translating user queries to visual encodings. Research has shown that using a natural language interface (NLI) is effective for these challenges because the user can simply talk to a computer capable of producing the graphs directly. In this paper, we discuss our intelligent assistant which processes speech and hand pointing gestures while also dealing with any number of visualizations on a large screen display. Evaluation of the system shows that it is capable of quickly producing visualizations. It also particularly effective at responding to less ambiguous queries, while in certain cases can handle ambiguous or complex queries. Exploring large datasets with visualizations makes gathering insights easier due to the ability to quickly identify patterns and compare trends between visualizations. However, for users unfamiliar with visualization, (Grammel, Tory, and Storey 2010) points out the steep learning curve necessary to translate high-level queries into visual representations. While popular tools such as Tableau can help facilitate the process, learning new user interfaces itself presents challenges that can overwhelm the user. Speech recognition and natural language processing, whose advances have in turn led to commercial success of natural language interfaces (NLIs) (e.g., virtual assistants such as Apple Siri, Amazon Alexa, and so on), have been a key focus of the research community in alleviating these challenges. In particular, various interactive visualization systems have been proposed (Cox et al. 2001; Reithinger et al. 2005; Sun et al. 2010; Setlur et al. 2016; Gao et al. 2015; Hoque et al. 2017; Yu and Silva 2019), that implement NLIs for visualization to help process verbal and nonverbal communication, effectively decouples the user interface from the user. In this paper, we discuss our own intelligent assistant that we claim provides a more supportive environment for data exploration relative to other systems; the user is able to use a large screen display to explore data, can interact with the large screen using free-forming NL as well as pointing gestures, and has the ability to manage multiple visualizations Copyright c © 2020, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. on the screen at once. Our contributions in this paper are: 1) Our assistant is implemented as a dialogue system (such systems are capable of having conversations with humans and have been successful in various domains, for example airline travel (Hemphill, Godfrey, and Doddington 1990; Budzianowski et al. 2018)). It is modeled after our own collected multimodal dialogue data (Kumar et al. 2016; 2017), capturing user interaction with a large screen display while tasked with exploring city crime data for Chicago (called CHICAGO-CRIME-VIS corpus). As a result, rather than making assumptions and enforcing NL templates, our approach is flexible to the different ways that user queries are spoken. Note that only 15% of our data contains such queries, which we refer to as actionable requests (AR) directly, while the remaining 85% of the spoken utterances do not specify actions to be taken by the system. 2) Our system is configured to operate on a large screen display, effectively allowing the user to preserve a sizable number of past visualizations on the screen. The user is able to quickly refer to any of these visualizations when forming subsequent ARs or analyzing recently constructed ones. 3) The system manages a dialogue history (DH) of past visualizations, which can be accessed in the future to retrieve visualizations that correspond to references made by the user. For example ”Show me the last heat map.” finds the most recent entry associated with map plots. 4) We conducted an extensive evaluation of the system with 20 subjects in a user study. The evaluation showed that the system is fast at responding to ARs and is also effective in generating satisfactory (according to user feedback) visualizations to ARs that have little ambiguity. One of the primary limitations observed by participants is the tendency by the system to take the meaning of an AR too literally, which can lead to a wrong understanding of the underlying intentions of the user and consequently a misleading visualization.