Toward multimodal notation for mathematics: Why and how

Notation is a tool of thought, recording and communicating concepts and activities related to a domain of knowledge. In the history of mathematical notation, written notation is usually considered. However, to be an effective thinking tool, notation must be properly perceived. Blind and partially sighted people run into difficulty in making, exploring and understanding mathematical concepts conceived for being represented in multi-dimensional space. In this paper, we capitalize on the multifaceted nature of digital symbols to define multimodal digital notation for graph structures that allows blind and partially sighted people to represent graph structures, reason about them and communicate their reasoning with sighted people as well as each other. As support tool for notation proposed we have designed a multimodal interactive system, in which haptic signals play a crucial role. An evolutionary prototype of the system has been developed and evaluated according to the star life cycle model.

[1]  Gerard Jounghyun Kim,et al.  Immersive authoring of Tangible Augmented Reality content: A user study , 2009, J. Vis. Lang. Comput..

[2]  Paul Curzon,et al.  People and Computers XX — Engage , 2007 .

[3]  Robert David Stevens,et al.  Principles for the Design of Auditory Interfaces to Present Complex Information to Blind People , 1996 .

[4]  David McGookin,et al.  Graph Builder: Constructing Non-visual Visualizations , 2007 .

[5]  G. Vanderheiden.  Nonvisual Alternative Display Techniques for Output from Graphics-Based Computers , 1989 .

[6]  Kenneth E. Iverson.  Notation as a tool of thought , 2007, ACM SIGAPL APL Quote Quad.

[7]  Alistair D. N. Edwards,et al.  An approach to the evaluation of assistive technology , 1996, Assets '96.

[8]  Daniela Fogli,et al.  Designing Customized and Tailorable Visual Interactive Systems , 2008, Int. J. Softw. Eng. Knowl. Eng..

[9]  Piero Mussio,et al.  Multimodal Exploration and Manipulation of Graph Structures , 2008, ICCHP.

[10]  Michael G. Strintzis,et al.  Haptic Rendering of Visual Data for the Visually Impaired , 2007, IEEE MultiMedia.

[11]  Piero Mussio,et al.  A multimodal interactive system to create and explore graph structures , 2008, CHI Extended Abstracts.

[12]  Benoît Encelle,et al.  LAMBDA: A European System to Access Mathematics with Braille and Audio Synthesis , 2006, ICCHP.

[13]  Thimoty Barbieri,et al.  Learning Math for Visually Impaired Users , 2008, ICCHP.

[14]  Christoph Schlieder,et al.  TeDUB: A System for Presenting and Exploring Technical Drawings for Blind People , 2002, ICCHP.

[15]  H. Rex Hartson,et al.  Developing user interfaces: ensuring usability through product & process , 1993 .

[16]  Alessandro Bianchi,et al.  Issues in design and implementation of multimedia software systems , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[17]  Vincent Hayward,et al.  Haptic interfaces and devices , 2004 .

[18]  T P Way,et al.  Automatic visual to tactile translation--Part I: Human factors, access methods, and image manipulation. , 1997, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[19]  Alistair D. N. Edwards,et al.  Lambda:: a multimodal approach to making mathematics accessible to blind students , 2006, Assets '06.

[20]  Piero Mussio,et al.  Toward haptic mathematics: why and how , 2008, AVI '08.

[21]  Daniela Fogli,et al.  Multi-Facet Design of Interactive Systems through Visual Languages , 2007 .

[22]  S. Brewer,et al.  CONTEXTUAL AUDIO IN HAPTIC GRAPH BROWSING , 2006 .

[23]  Stephen A. Brewster,et al.  MultiVis: improving access to visualisations for visually impaired people , 2006, CHI Extended Abstracts.

[24]  Fernando Ferri.  Visual Languages for Interactive Computing: Definitions and Formalizations , 2007 .