Thinking with Diagrams

What properties, if any, distinguish graphical representations from linguistic representations? This paper looks for answers in the literature of philosophy, logic, artificial intelligence, and cognitive psychology, and extracts seven alternative binary classifications of representations that may characterize the graphic-linguistic boundary. We assess each alternative by two standards: (a) whether it extensionally fits the graphic-linguistic distinction, and (b) how far it explains the properties commonly attributed to graphic representations but not

[1]  Yvonne Rogers,et al.  How can interactive multimedia facilitate learning , 1998 .

[2]  Richard Lowe,et al.  Selectivity in diagrams: reading beyond the lines , 1994 .

[3]  Daniel M. Herbert,et al.  Architectural Study Drawings , 1993 .

[4]  D. Kibler,et al.  Exploring the Episodic Structure of Algebra Story Problem Solving , 1989 .

[5]  Paul Brna,et al.  Collaborative virtual learning environments for concept learning , 1999 .

[6]  Ernest Davis,et al.  Representations of commonsense knowledge , 2014, notThenot Morgan Kaufmann series in representation and reasoning.

[7]  A. Collins,et al.  Situated Cognition and the Culture of Learning , 1989 .

[8]  M. de Rijke,et al.  The Modal Logic of Inequality , 1992, J. Symb. Log..

[9]  Oliver Lemon,et al.  Spatial logic and the complexity of diagrammatic reasoning , 1997 .

[10]  Herbert A. Simon,et al.  Why a diagram is (sometimes) worth 10, 000 word , 1987 .

[11]  Efficacy of Diagrammatic Reasoning , 1999 .

[12]  Dimitris Papadias,et al.  Topological Inference , 1995, IJCAI.

[13]  M. Aurnague,et al.  A three-level approach to the semantics of space , 1993 .

[14]  Paul Brna,et al.  Modelling students collaborating while learning about energy , 1997, J. Comput. Assist. Learn..

[15]  Ellen Yi-Luen Do.  Computability of Design Diagrams - An Empirical Study of Diagram Conventions in Design , 1997 .

[16]  Baruch Schwarz,et al.  Measuring Integration of Information in Multirepresentational Software , 1993, Interact. Learn. Environ..

[17]  George W. Furnas,et al.  Reasoning With Diagrams Only , 2002 .

[18]  Alan F. Blackwell,et al.  Metacognitive theories of visual programming: what do we think we are doing? , 1996, Proceedings 1996 IEEE Symposium on Visual Languages.

[19]  Oliver Lemon,et al.  On the Incompleteness of Modal Logics of Space: Advancing Complete Modal Logics of Place , 1996, Advances in Modal Logic.

[20]  Richard K. Lowe.  Background knowledge and the construction of a situational representation from a diagram , 1996 .

[21]  Judy S. DeLoache,et al.  When a picture is not worth a thousand words: Young children's understanding of pictures and models , 1992 .

[22]  O. N. Romaniv,et al.  The informational component in systems of corrosion diagnostics for engineering equipment , 1991 .

[23]  Paul Brna,et al.  Reasoning with external representations: supporting the stages of selection, construction and use , 1993, INTERCHI Adjunct Proceedings.

[24]  L. Rips.  The Psychology of Proof , 1994 .

[25]  P. Johnson-Laird.  Mental models , 1989 .

[26]  H. Levesque.  Logic and the complexity of reasoning , 1988 .

[27]  Kirsten N. Whitley.  Visual Programming Languages and the Empirical Evidence For and Against , 1997, J. Vis. Lang. Comput..

[28]  Marian Petre.  Shifts in Reasoning about Software and Hardware Systems: Must Operational Models Underpin Declarative Ones? , 1991, PPIG.

[29]  M. Braine.  On the Relation Between the Natural Logic of Reasoning and Standard Logic. , 1978 .

[30]  Lenhart K. Schubert.  Extending The Expressive Power Of Semantic Networks , 1976, IJCAI.

[31]  K. Stenning,et al.  Image and Language in Human Reasoning: A Syllogistic Illustration , 1997, Cognitive Psychology.

[32]  Paul J. Feltovich,et al.  Categorization and Representation of Physics Problems by Experts and Novices , 1981, Cogn. Sci..

[33]  Gudula Retz-Schmidt,et al.  Various Views on Spatial Prepositions , 1988, AI Mag..

[34]  Muneo Kitajima,et al.  A comprehension-based model of exploration , 1996, Hum. Comput. Interact..

[35]  Craig Zimring,et al.  Supporting Case-Study Use in Design Education: A Computational Case-Based Design Aid for Architecture , 1995 .

[36]  I. Gordon.  Theories of Visual Perception , 1989 .

[37]  Marian Petre,et al.  Usability Analysis of Visual Programming Environments: A 'Cognitive Dimensions' Framework , 1996, J. Vis. Lang. Comput..

[38]  Thomas R. Ioerger,et al.  The Manipulation of Images to Handle Indeterminacy in Spatial Reasoning , 1994, Cogn. Sci..

[39]  Nelson David Ludlow,et al.  Pictorial representation of text : converting text to pictures , 1993 .

[40]  V. Goel.  Sketches of thought , 1995 .

[41]  Donald A. Norman,et al.  Accretion, tuning and restructuring: Three modes of learning , 1976 .

[42]  Nick Hammond,et al.  Argumentation-based design rationale: what use at what cost? , 1994, Int. J. Hum. Comput. Stud..

[43]  Michelene T. H. Chi,et al.  Self-Explanations: How Students Study and Use Examples in Learning To Solve Problems. Technical Report No. 9. , 1987 .

[44]  Oliver Lemon,et al.  On the Insufficiency of Linear Diagrams for Syllogisms , 1998, Notre Dame J. Formal Log..

[45]  Jiajie Zhang,et al.  Representations in Distributed Cognitive Tasks , 1994, Cogn. Sci..

[46]  Herbert A. Simon,et al.  Why a Diagram is (Sometimes) Worth Ten Thousand Words , 1987, Cogn. Sci..

[47]  Margaret M. Burnett,et al.  Is it easier to write matrix manipulation programs visually or textually? An empirical study , 1993, Proceedings 1993 IEEE Symposium on Visual Languages.

[48]  D E Egan,et al.  Chunking in recall of symbolic drawings , 1979, Memory & cognition.

[49]  Herbert A. Simon,et al.  SCIENTIFIC DISCOVERY AND CREATIVE REASONING WITH DIAGRAMS , 2003 .

[50]  Brian V. Funt,et al.  Problem-Solving with Diagrammatic Representations , 1980, Artif. Intell..

[51]  Patrick Olivier,et al.  Automatic Depiction of Spatial Descriptions , 1994, AAAI.

[52]  Thomas P. Moran,et al.  Analogy considered harmful , 1982, CHI '82.

[53]  L. Cronbach,et al.  Aptitudes and instructional methods: A handbook for research on interactions , 1977 .

[54]  B. Lawson.  Design in mind , 1994 .

[55]  R. Giere.  Explaining Science: A Cognitive Approach , 1991 .

[56]  Ian Pratt-Hartmann,et al.  A Complete Axiom System for Polygonal Mereotopology of the Real Plane , 1998, J. Philos. Log..

[57]  Bill Curtis,et al.  Experimental evaluation of software documentation formats , 1989, J. Syst. Softw..

[58]  Akihiro Kashihara.  A Self-Explanation Assistance with Diagram Tailoring , 1996 .

[59]  Herbert A. Simon,et al.  Situated Action: A Symbolic Interpretation , 1993, Cogn. Sci..

[60]  Paul Laseau.  Graphic Problem Solving for Architects and Designers , 1986 .

[61]  Oliver Lemon,et al.  Semantical Foundations of Spatial Logics , 1996, KR.

[62]  Kincho H. Law,et al.  Qualitative Structural Analysis Using Diagrammatic Reasoning , 1995, IJCAI.

[63]  Patrick Oliver,et al.  Representation and Processing of Spatial Expressions , 1998 .

[64]  K. Holyoak.  Mental representations. , 1982, Science.

[65]  Margaret M. Burnett,et al.  Representation Design Benchmarks: A Design-Time Aid for VPL Navigable Static Representations , 1997, J. Vis. Lang. Comput..

[66]  Oliver Lemon,et al.  Putting channels on the map: a channel-theoretic semantics of maps? , 1999 .

[67]  Atsushi Shimojima,et al.  ON THE EFFICACY OF REPRESENTATION , 1996 .

[68]  P.C.-H. Cheng.  Thinking, expertise and diagrams that encode laws , 1996 .

[69]  Yvonne Rogers,et al.  External cognition: how do graphical representations work? , 1996, Int. J. Hum. Comput. Stud..

[70]  Michael F. Schober,et al.  Speakers, addressees, and frames of reference: Whose effort is minimized in conversations about locations? , 1995 .

[71]  Valentin Goranko,et al.  Modal logic with names , 1993, J. Philos. Log..

[72]  Richard Cox.  Analytical reasoning with multiple external representations , 1996 .

[73]  Janellen Huttenlocher,et al.  Constructing spatial images: A strategy in reasoning. , 1968 .

[74]  Marian Petre,et al.  Issues Governing the Suitability of Programming Languages for Programming Tasks , 1988, BCS HCI.

[75]  山田 篤.  Studies on spatial description understanding based on geometric constraints satisfaction , 1993 .

[76]  Richard Lowe,et al.  Diagram prediction and higher order structures in mental representation , 1994 .

[77]  Sun-Joo Shin,et al.  The logical status of diagrams , 1995 .

[78]  Ephraim P. Glinert,et al.  Visual Programming Environments: Paradigms and Systems , 1990 .

[79]  Ellen Yi-Luen Do,et al.  The right tool at the right time: investigation of freehand drawing as an interface to knowledge-based design tools , 1998 .

[80]  Wolfgang Schnotz,et al.  How do successful and unsuccessful learners use texts and graphics , 1993 .

[81]  Eugene S. Ferguson,et al.  Engineering and the Mind's Eye , 1994 .

[82]  Richard Tobin,et al.  Assigning Information to Modalities: Comparing Graphical Treatments of the Syllogism , 1997 .

[83]  Zenon Kulpa,et al.  DIAGRAMMATIC REPRESENTATION AND REASONING , 1994 .

[84]  K. Jon Barwise,et al.  Heterogeneous Reasoning , 1993, International Conference on Conceptual Structures.

[85]  Bernhard Nebel,et al.  Computational Properties of Qualitative Spatial Reasoning: First Results , 1995, KI.

[86]  S. Levinson.  Frames of reference and Molyneux's question: Cross-linguistic evidence , 1996 .

[87]  G. Stiny.  Introduction to Shape and Shape Grammars , 1980 .

[88]  Jon Oberlander,et al.  Individual differences in proof structures following multimodal logic teaching , 1996 .

[89]  Hector J. Levesque.  Making Believers out of Computers , 1989 .

[90]  John T. Stasko,et al.  Do algorithm animations assist learning?: an empirical study and analysis , 1993, INTERCHI.

[91]  Richard K. Lowe.  Search Strategies and Inference in the Exploration of Scientific Diagrams , 1989 .

[92]  Volker Haarslev.  Formal semantics of visual languages using spatial reasoning , 1995, Proceedings of Symposium on Visual Languages.

[93]  Donald A. Norman,et al.  Cognition in the Head and in the World: An Introduction to the Special Issue on Situated Action , 1993, Cogn. Sci..

[94]  Longin Jan Latecki,et al.  On Hybrid Reasoning for Processing Spatial Expressions , 1992, ECAI.

[95]  Alan F. Blackwell,et al.  A glimpse of expert programmers' mental imagery , 1997, ESP '97.

[96]  Ellen Yi-Luen.  What ' s in a diagram that a computer should understand , 1995 .

[97]  Alan F. Blackwell,et al.  Cognitive Questions in Software Visualisation , 1997 .

[98]  W. Lewis Johnson,et al.  Mixed-Initiative Interaction between Pedagogical Agents and Students in Virtual Environments , 1997 .

[99]  Ben Shneiderman,et al.  Experimental investigations of the utility of detailed flowcharts in programming , 1977, CACM.

[100]  Peter C.-H. Cheng,et al.  Functional Roles for the Cognitive Analysis of Diagrams in Problem Solving , 2002 .

[101]  Eric Hammer,et al.  Logic and Visual Information , 1995 .

[102]  W. Kintsch.  The role of knowledge in discourse comprehension: a construction-integration model. , 1988, Psychological review.

[103]  David L. Waltz,et al.  Visual Analog Representations for Natural Languages Understanding , 1979, IJCAI.

[104]  Ted G. Lewis,et al.  Visual Object-Oriented Programming: Concepts and Environments , 1995 .

[105]  George Englebretsen,et al.  Linear Diagrams for Syllogisms (with Relatitonals) , 1991, Notre Dame J. Formal Log..

[106]  Ron Weber,et al.  Structured tools and conditional logic: an empirical investigation , 1986, CACM.

[107]  Daniel L. Schwartz,et al.  Analog Imagery in Mental Model Reasoning: Depictive Models , 1996, Cognitive Psychology.

[108]  H. Gelernter,et al.  Realization of a geometry theorem proving machine , 1995, IFIP Congress.

[109]  Kenneth M. Kahn,et al.  ToonTalk - An Animated Programming Environment for Children , 1996, J. Vis. Lang. Comput..

[110]  M. Miyata.  THEORIES OF REPRESENTATION IN NAZI'S STAATSLEHRE , 1958 .

[111]  Paul Brna.  Can't see the words for the tree: interpretation and graphical representations , 1996 .

[112]  T. Mcdougal,et al.  A recognition model of geometry theorem-proving , 1992 .

[113]  Ed Baroth,et al.  Visual programming in the real world , 1995 .

[114]  Bernard Meltzer,et al.  Analogical Representations of Naive Physics , 1989, Artif. Intell..

[115]  Richard M. Young,et al.  A dual-space model of iteratively deepening exploratory learning , 1996, Int. J. Hum. Comput. Stud..

[116]  D. A. Schon.  Designing as reflective conversation with the materials of a design situation , 1992 .

[117]  Jon Oberlander,et al.  A Cognitive Theory of Graphical and Linguistic Reasoning: Logic and Implementation , 1995, Cogn. Sci..

[118]  Alexander Repenning,et al.  Agentsheets: A Medium for Creating Domain-Oriented Languages , 1995, Computer.

[119]  John R. Anderson,et al.  Abstract Planning and Perceptual Chunks: Elements of Expertise in Geometry , 1990, Cogn. Sci..

[120]  R. Mayer,et al.  For whom is a picture worth a thousand words? Extensions of a dual-coding theory of multimedia learning. , 1994 .

[121]  Christos H. Papadimitriou,et al.  Computational complexity , 1993 .

[122]  Bonnie Grossen,et al.  Diagramming a Logic Strategy: Effects on Difficult Problem Types and Transfer , 1990 .

[123]  Paul Brna,et al.  What is the Value of Virtual Reality for Conceptual Learning? Towards a Theoretical Framework , 1996 .

[124]  Alan F. Blackwell,et al.  Metaphor in diagrams , 1998 .

[125]  Judith Good.  Programming paradigms, information types and graphical representations : empirical investigations of novice program comprehension , 1999 .

[126]  Michael J. Baker,et al.  Flexibly structuring the interaction in a CSCL environment , 1996 .

[127]  Richard Cox,et al.  The effect of graphical and sentential logic teaching on spontaneous external representation , 1995 .

[128]  Marian Petre,et al.  When Visual Programs are Harder to Read than Textual Programs , 1992 .

[129]  Nancy Cunniff,et al.  Graphical vs. textual representation: an empirical study of novices' program comprehension , 1987 .

[130]  Herbert H. Clark,et al.  Linguistic processes in deductive reasoning. , 1969 .

[131]  Richard Lowe.  Pictorial information design for schools , 1995 .

[132]  N. Hari Narayanan,et al.  IMAGERY: COMPUTATIONAL AND COGNITIVE PERSPECTIVES , 1993 .

[133]  Seymour Papert,et al.  Mindstorms: Children, Computers, and Powerful Ideas , 1981 .

[134]  Brad A. Myers,et al.  Visual Programming in a Visual Domain: A Case Study of Cognitive Dimensions , 1994, BCS HCI.

[135]  G. Salomon.  Interaction of media, cognition and learning , 1979 .

[136]  David J. Gilmore,et al.  Comprehension and Recall of Miniature Programs , 1984, Int. J. Man Mach. Stud..