The automated understanding of simple bar charts

While identifying the intention of an utterance has played a major role in natural language understanding, this work is the first to extend intention recognition to the domain of information graphics. A tenet of this work is the belief that information graphics are a form of language. This is supported by the observation that the overwhelming majority of information graphics from popular media sources appear to have some underlying goal or intended message. As Clark noted, language is more than just words. It is any ''signal'' (or lack of signal when one is expected), where a signal is a deliberate action that is intended to convey a message (Clark, 1996 [15]). As a form of language, information graphics contain communicative signals that can be used in a computational system to identify the message that the graphic conveys. We identify the communicative signals that appear in simple bar charts, and present an implemented Bayesian network methodology for reasoning about these signals and hypothesizing a bar chart's intended message. Once the message conveyed by an information graphic has been inferred, it can then be used to facilitate access to this information resource for a variety of users, including 1) users of digital libraries, 2) visually impaired users, and 3) users of devices where graphics are impractical or inaccessible.

[1]  Peng Wu,et al.  Segmenting Line Graphs Into Trends , 2010, IC-AI.

[2]  Johanna D. Moore,et al.  Describing Complex Charts in Natural Language: A Caption Generation System , 1998, CL.

[3]  Wenke Lee,et al.  Attack plan recognition and prediction using causal networks , 2004, 20th Annual Computer Security Applications Conference.

[4]  K. Holmqvist,et al.  Reading Information Graphics: the Role of Spatial Contiguity and Dual Attentional Guidance , 2022 .

[5]  Michael Johnston,et al.  MATCHkiosk: A Multimodal Interactive City Guide , 2004, ACL.

[6]  K. Korta,et al.  Plan Recognition in Natural Language Dialogue , 1994 .

[7]  Bonnie E. John,et al.  Toward an Engineering Model of Stimulus-Response Compatibility , 1990 .

[8]  Lynn Lambert.  Recognizing complex discourse acts: a tripartite plan-based model of dialogue , 1993 .

[9]  Raj M Ratwani,et al.  Thinking graphically: Connecting vision and cognition during graph comprehension. , 2008, Journal of experimental psychology. Applied.

[10]  Gerald L. Lohse,et al.  A Cognitive Model for Understanding Graphical Perception , 1993, Hum. Comput. Interact..

[11]  Hendrik Van Brussel,et al.  Bayesian plan recognition and shared control under uncertainty: assisting wheelchair drivers by tracking fine motion paths , 2007, 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[12]  Robert P. Goldman,et al.  A probabilistic plan recognition algorithm based on plan tree grammars , 2009, Artif. Intell..

[13]  James F. Allen.  A plan-based approach to speech act recognition , 1979 .

[14]  Richard Burns,et al.  Modeling Relative Task Effort for Grouped Bar Charts , 2009 .

[15]  S. Kosslyn.  Understanding charts and graphs , 1989 .

[16]  Guy Lapalme,et al.  Intentions in the Coordinated Generation of Graphics and Text from Tabular Data , 2000, Knowledge and Information Systems.

[17]  Vibhu O. Mittal.  Visual Prompts and Graphical Design: A Framework for Exploring the Design Space of 2-D Charts and Graphs , 1997, AAAI/IAAI.

[18]  Steven F. Roth,et al.  Mapping communicative goals into conceptual tasks to generate graphics in discourse , 2001, Knowl. Based Syst..

[19]  Ingrid Zukerman,et al.  Towards a Bayesian Model for Keyhole Plan Recognition in Large Domains , 1997 .

[20]  Eric Horvitz,et al.  A computational architecture for conversation , 1999 .

[21]  Edmund H. Durfee,et al.  The Automated Mapping of Plans for Plan Recognition , 1994, AAAI.

[22]  Daniel L. Chester,et al.  Getting Computers to See Information Graphics So Users Do Not Have to , 2005, ISMIS.

[23]  Nancy Green,et al.  A Model of Perceptual Task Effort for Bar Charts and its Role in Recognizing Intention , 2006, User Modeling and User-Adapted Interaction.

[24]  Nancy Green,et al.  Incorporating Perceptual Task Effort into the Recognition of Intention in Information Graphics , 2004, Diagrams.

[25]  C. Raymond Perrault,et al.  Analyzing Intention in Utterances , 1986, Artif. Intell..

[26]  Michael P. Wellman,et al.  Probabilistic State-Dependent Grammars for Plan Recognition , 2000, UAI.

[27]  Stephen M. Kosslyn,et al.  Elements of graph design , 1993 .

[28]  Martha E. Pollack,et al.  Autominder: an intelligent cognitive orthotic system for people with memory impairment , 2003, Robotics Auton. Syst..

[29]  Herbert A. Simon,et al.  Why a Diagram is (Sometimes) Worth Ten Thousand Words , 1987, Cogn. Sci..

[30]  C. Raymond Perrault,et al.  A Plan-Based Analysis of Indirect Speech Act , 1980, CL.

[31]  Kees van Deemter,et al.  High-level authoring of illustrated documents , 2003, Natural Language Engineering.

[32]  Peter Poller,et al.  Situated Delegation-Oriented Multimodal Presentation in SmartKom , 2002 .

[33]  Nate Blaylock,et al.  Generating Artificial Corpora for Plan Recognition , 2005, User Modeling.

[34]  Robert P. Futrelle,et al.  Summarization of Diagrams in Documents , 1999 .

[35]  Stephanie Elzer Schwartz,et al.  Exploiting Evidence Analysis in Plan Recognition , 2007, User Modeling.

[36]  Robert P. Goldman,et al.  Probabilistic Plan Recognition for Hostile Agents , 2001, FLAIRS Conference.

[37]  Susan Bell Trickett,et al.  Toward a Comprehensive Model of Graph Comprehension: Making the Case for Spatial Cognition , 2006, Diagrams.

[38]  H. Grice.  Utterer's meaning and intentions , 1969 .

[39]  Deb Roy,et al.  Situated Language Understanding as Filtering Perceived Affordances , 2007, Cogn. Sci..

[40]  H. Herne,et al.  How to Lie with Statistics , 1973 .

[41]  Deb Roy,et al.  Probabilistic grounding of situated speech using plan recognition and reference resolution , 2005, ICMI '05.

[42]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems , 1988 .

[43]  L. C. van der Gaag,et al.  Building probabilistic networks: Where do the numbers come from? - a guide to the literature , 2000 .

[44]  Allen Newell,et al.  The psychology of human-computer interaction , 1983 .

[45]  Martha E. Pollack,et al.  A Model of Plan Inference That Distinguishes Between the Beliefs of Actors and Observers , 1986, ACL.

[46]  Moo-Song Lee.  Statistics, the Conceptual Approach. Gudmund R. Iversen and Mary Gergen, Springer, New York, 1997. No. of pages: xviii+735. Price: $54.95. ISBN 0-387-94610-1 , 1999 .

[47]  James A. Landay,et al.  The Mobile Sensing Platform: An Embedded Activity Recognition System , 2008, IEEE Pervasive Computing.

[48]  M. Corio,et al.  Generation of texts for information graphics , 1999 .

[49]  Andreas Stolcke,et al.  Prosody-based automatic detection of annoyance and frustration in human-computer dialog , 2002, INTERSPEECH.

[50]  Wolfgang Wahlster,et al.  Plan-Based Integration of Natural Language and Graphics Generation , 1993, Artif. Intell..

[51]  Robert P. Futrelle,et al.  Constraint based vectorization , 1989, ICS '89.

[52]  Robert P. Goldman,et al.  A Bayesian Model of Plan Recognition , 1993, Artif. Intell..

[53]  Ingrid Zukerman,et al.  Exploring and Exploiting the Limited Utility of Captions in Recognizing Intention in Information Graphics , 2005, ACL.

[54]  Wendy G. Lehnert,et al.  Strategies for Natural Language Processing , 1982 .

[55]  Milind Tambe,et al.  Monitoring Teams by Overhearing: A Multi-Agent Plan-Recognition Approach , 2002, J. Artif. Intell. Res..

[56]  Robert P. Goldman,et al.  Plan recognition in intrusion detection systems , 2001, Proceedings DARPA Information Survivability Conference and Exposition II. DISCEX'01.

[57]  William J. Wilson,et al.  Statistics: The Conceptual Approach , 2011 .

[58]  Candace L. Sidner,et al.  Attention, Intentions, and the Structure of Discourse , 1986, CL.

[59]  Josef Ruppenhofer,et al.  FrameNet II: Extended theory and practice , 2006 .

[60]  Chris Mellish,et al.  Choosing the content of textual summaries of large time-series data sets , 2006, Natural Language Engineering.

[61]  Ingrid Zukerman,et al.  Bayesian Models for Keyhole Plan Recognition in an Adventure Game , 2004, User Modeling and User-Adapted Interaction.

[62]  John A. Bateman,et al.  Towards Constructive Text, Diagram, and Layout Generation for Information Presentation , 2001, Computational Linguistics.

[63]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[64]  Peter C.-H. Cheng,et al.  Modeling the Effect of Task and Graphical Representation on Response Latency in a Graph Reading Task , 2003, Hum. Factors.

[65]  C. Raymond Perrault,et al.  Beyond Question-Answering. , 1981 .

[66]  Johanna D. Moore,et al.  AutoBrief: an experimental system for the automatic generation of briefings in integrated text and information graphics , 2004, Int. J. Hum. Comput. Stud..

[67]  Nikos Fakotakis,et al.  MeteoBayes: Effective Plan Recognition in a Weather Dialogue System , 2007, IEEE Intelligent Systems.

[68]  Marek J. Druzdzel,et al.  Building Probabilistic Networks: "Where Do the Numbers Come From?" Guest Editors Introduction , 2000, IEEE Trans. Knowl. Data Eng..