Text Detection and Translation from Natural Scenes

Abstract : The authors present a system for automatic extraction and interpretation of signs from a natural scene. The system is capable of capturing images, detecting and recognizing signs, and translating them into a target language. The translation can be displayed on a hand-held wearable display or a head-mounted display. It can also be synthesized as a voice output message over the earphones. The paper addresses challenges in automatic sign extraction and translation, describes methods for automatic sign extraction, and extends example-based machine translation technology for sign translation. The authors use a user-centered approach in system development that takes advantage of human intelligence and leverages human capabilities. They are currently working on Chinese sign translation. So far, they have developed a prototype system that can recognize Chinese signs from a video camera and then translate them either into English text or a voice stream. They have built a database containing about 800 Chinese signs for development and evaluation. The authors hope that the sign translation, in conjunction with spoken language translation, will help international tourists overcome language barriers. The technology could also help a visually handicapped person increase his or her environmental awareness.