An interactive and intuitive stem accessibility system for the blind and visually impaired

We present an intuitive and interactive platform to make complex STEM (Science Technology Engineering and Mathematics) educational materials accessible to blind and visually impaired people using a mini-hyperbraille device with almost no loss of information as compared to printed materials. We have come up with a novel way to increase the effective resolution of the braille device by adding a mechanical XY gantry. Using this XY gantry enables us to create an ultrahigh resolution, larger surface braille device without paying the prohibitive price charged for the bigger hyper Braille displays available in the market. In addition to that, to further augment usability and intuitiveness of our system, we have integrated a nod-ring, which is a tiny finger-worn device for supporting additional hand gestures such as zoom in and out. Previous studies have shown that the use of zoom and pan can increase usability and improve the understanding of tactile graphics. Along with zooming and panning, our system uses vibrating patterns, rhythmic motions, synthetic voice and synchronized voiced-vibrations to convey information to blind users in an intuitive way. We also implemented a touch gesture recognition framework on our touch enabled braille device. Using these touch gestures and a high quality synthetic voice, we have developed a highly responsive system for providing voice annotations of the graphics content. An important contribution of this work is the implementation of a high-quality system for automatic transcription of STEM (including difficult math Nemeth translations) books into braille. Learning resources for blind people are quite sparse and transcription of STEM material is a very expensive and time consuming process. Using our automated transcription platform it is easy, fast and inexpensive for publishers to transcribe STEM books into braille books. The scope of this automated transcription platform is not only limited to STEM books but it can be used for transcription of any content/book/web-page available on-line.

[1]  Jack M Loornis.  Tactile pattern perception , 1981 .

[2]  T. Martin.  American Printing House for the Blind , 2014 .

[3]  N. Di Spigna,et al.  Application of EAP materials toward a refreshable Braille display , 2009, Smart Structures and Materials + Nondestructive Evaluation and Health Monitoring.

[4]  Yong Wang,et al.  A compact electroactive polymer actuator suitable for refreshable Braille display , 2008 .

[5]  J C Craig,et al.  Vibrotactile pattern perception: extraordinary observers. , 1977, Science.

[6]  Gordon G. Wallace,et al.  Actuated Pins for Braille Displays , 2009 .

[7]  R. Klatzky,et al.  Haptic perception: A tutorial , 2009, Attention, perception & psychophysics.

[8]  R. Klatzky,et al.  Haptic identification of objects and their depictions , 1993, Perception & psychophysics.

[9]  Binbin Xi,et al.  Ionic liquids and polypyrrole helix tubes: bringing the electronic Braille screen closer to reality , 2003, SPIE Smart Structures and Materials + Nondestructive Evaluation and Health Monitoring.

[10]  J C Craig,et al.  Tactile letter recognition: Pattern duration and modes of pattern generation , 1981, Perception & psychophysics.

[11]  T. Someya,et al.  Sheet-Type Braille Displays by Integrating Organic Field-Effect Transistors and Polymeric Actuators , 2007, IEEE Transactions on Electron Devices.

[12]  John A. Gardner,et al.  Scientific Diagrams Made Easy with IVEOTM , 2006, ICCHP.

[13]  R. Klatzky,et al.  Visual mediation and the haptic recognition of two-dimensional pictures of common objects , 1990, Perception & psychophysics.

[14]  Yoseph Bar-Cohen,et al.  Refreshable Braille displays using EAP actuators , 2010, Smart Structures and Materials + Nondestructive Evaluation and Health Monitoring.

[15]  Ravi Rastogi,et al.  Intuitive Tactile Zooming for Graphics Accessed by Individuals Who are Blind and Visually Impaired , 2013, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[16]  R. Klatzky,et al.  Similarity of Tactual and Visual Picture Recognition with Limited Field of View , 1991, Perception.

[17]  Louis H. Goldish,et al.  The Optacon: A Valuable Device for Blind Persons , 1974 .

[18]  Vincent Hayward,et al.  Adaptive level of detail in dynamic, refreshable tactile graphics , 2012, 2012 IEEE Haptics Symposium (HAPTICS).

[19]  Christoph Schlieder,et al.  TeDUB: A System for Presenting and Exploring Technical Drawings for Blind People , 2002, ICCHP.

[20]  Jane N. Erin,et al.  Use of the Talking Tactile Tablet in Mathematics Testing , 2003 .

[21]  Marc Alexa,et al.  How do humans sketch objects? , 2012, ACM Trans. Graph..

[22]  Bin Chen,et al.  Large-strain, rigid-to-rigid deformation of bistable electroactive polymers , 2009 .

[23]  Stephen A. Brewster,et al.  A generic approach for augmenting tactile diagrams with spatial non-speech sounds , 2002, CHI Extended Abstracts.

[24]  Charles Lenay,et al.  Design of a Haptic Zoom: levels and steps , 2007, Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC'07).

[25]  M A Heller,et al.  Tactual picture identification by blind and sighted people: Effects of providing categorical information , 1996, Perception & psychophysics.

[26]  Nicholas A Giudice,et al.  Journal of Experimental Psychology : Learning , Memory , and Cognition Functional Equivalence of Spatial Images From Touch and Vision : Evidence From Spatial Updating in Blind and Sighted Individuals , 2011 .