Enabling Seamless Access to Digital Graphical Contents for Visually Impaired Individuals via Semantic-Aware Processing

Vision is one of the main sources through which people obtain information from the world, but unfortunately, visually-impaired people are partially or completely deprived of this type of information. With the help of computer technologies, people with visual impairment can independently access digital textual information by using text-to-speech and text-to-Braille software. However, in general, there still exists a major barrier for people who are blind to access the graphical information independently in real-time without the help of sighted people. In this paper, we propose a novel multi-level and multi-modal approach aiming at addressing this challenging and practical problem, with the key idea being semantic-aware visual-to-tactile conversion through semantic image categorization and segmentation, and semantic-driven image simplification. An end-to-end prototype system was built based on the approach. We present the details of the approach and the system, report sample experimental results with realistic data, and compare our approach with current typical practice.

[1]  Oded Maron,et al.  Learning from Ambiguity , 1998 .

[2]  J. Kennedy,et al.  Exploring pictures tactually , 1980, Nature.

[3]  J M Kennedy,et al.  Close Encounters: Details Veto Depth from Shadows , 2000, Science.

[4]  Linda Pring,et al.  Tactual picture recognition in congenitally blind and sighted children , 1989 .

[5]  J M Kennedy,et al.  Depictions of motion devised by a blind person , 2000, Psychonomic bulletin & review.

[6]  Hideyuki Ando,et al.  The nail-mounted tactile display for the behavior modeling , 2002, SIGGRAPH '02.

[7]  Robert S. Wall,et al.  Training and Availability of Braille Transcribers in the United States , 2002 .

[8]  Stéphane Canu,et al.  Technology and perception: the contribution of sensory substitution systems , 1997, Proceedings Second International Conference on Cognitive Technology Humanizing the Information Age.

[9]  James C. Bliss,et al.  Optical-to-Tactile Image Conversion for the Blind , 1970 .

[10]  T P Way,et al.  Automatic visual to tactile translation--Part II: Evaluation of the TACTile Image Creation System. , 1997, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[11]  Masahiko Inami,et al.  The SmartTool: a system for augmented reality of haptics , 2002, Proceedings IEEE Virtual Reality 2002.

[12]  John M Kennedy,et al.  Haptic Pictures: Fit Judgments Predict Identification, Recognition Memory, and Confidence , 2002, Perception.

[13]  Ronald Hinton.  First introduction to tactiles , 1991 .

[14]  P Bach-y-Rita,et al.  Transmission of pictorial information through the skin. , 1973, Advances in biological and medical physics.

[15]  M Akamatsu,et al.  Please Scroll down for Article Ergonomics a Comparison of Tactile, Auditory, and Visual Feedback in a Pointing Task Using a Mouse-type Device , 2022 .

[16]  Denis G. Pelli,et al.  Close encounters: Details veto depth from shadows [6] (multiple letters) , 2000 .

[17]  James Ze Wang,et al.  SIMPLIcity: Semantics-Sensitive Integrated Matching for Picture LIbraries , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[18]  James Ze Wang,et al.  Automatic Linguistic Indexing of Pictures by a Statistical Modeling Approach , 2003, IEEE Trans. Pattern Anal. Mach. Intell..

[19]  Yixin Chen,et al.  Image Categorization by Learning and Reasoning with Regions , 2004, J. Mach. Learn. Res..

[20]  Igor Juricevic,et al.  Haptics and Projection: Drawings by Tracy, a Blind Adult , 2003, Perception.

[21]  Yvette Hatwell,et al.  Chapter 15. The tactile reading of maps and drawings, and the access of blind people to works of art , 2003 .

[22]  A. D. Heyes.  Human navigation by sound , 1983 .

[23]  Cuihua Li,et al.  BUILDING DETECTION AND RECOGNITION VIA THE IMPROVED HOUGH TRANSFORM , 2004 .

[24]  Richard E. Ladner,et al.  Automating tactile graphics translation , 2005, Assets '05.

[25]  Peter B. L. Meijer,et al.  An experimental system for auditory image representations , 1992, IEEE Transactions on Biomedical Engineering.

[26]  Sally A. Goldman,et al.  Multiple-Instance Learning of Real-Valued Data , 2001, J. Mach. Learn. Res..

[27]  Kenneth J. Kokjer,et al.  The Information Capacity of the Human Fingertip , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[28]  Baoxin Li,et al.  Detecting and classifying frontal, back and profile views of humans , 2007, VISAPP.

[29]  Tomás Lozano-Pérez,et al.  A Framework for Multiple-Instance Learning , 1997, NIPS.

[30]  J M Kennedy,et al.  Children's tactual exploration and copying without vision , 2001, International journal of rehabilitation research. Internationale Zeitschrift fur Rehabilitationsforschung. Revue internationale de recherches de readaptation.

[31]  J M Kennedy,et al.  Perspective taking, pictures, and the blind , 1990, Perception & psychophysics.

[32]  L. Merabet,et al.  What blindness can tell us about seeing again: merging neuroplasticity and neuroprostheses , 2005, Nature Reviews Neuroscience.

[33]  Baoxin Li,et al.  Multiple Class Multiple-Instance Learning and its Application to Image Categorization , 2007, Int. J. Image Graph..

[34]  A. Corn,et al.  Production of Textbooks and Instructional Materials in the United States , 2002 .

[35]  D. Burger.  Improved access to computers for the visually handicapped: new prospects and principles , 1994 .

[36]  Qi Zhang,et al.  EM-DD: An Improved Multiple-Instance Learning Technique , 2001, NIPS.

[37]  T P Way,et al.  Automatic visual to tactile translation--Part I: Human factors, access methods, and image manipulation. , 1997, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.