Snap and Translate Using Windows Phone

We have developed a prototype of a mobile app called "Snap and Translate" on "Windows Phone 7". A person who is reading an English menu/sign and wants a Chinese translation of an English word or phrase or paragraph can use a Windows Phone to snap an image of the text, tap the word or swipe the phrase or circle the paragraph with a finger, and get a Chinese translation displayed on the screen of the phone. This is enabled by seamless integration of three Microsoft technologies: intelligent text extraction, OCR, and machine translation based on a client-plus-cloud architecture. The current prototype also supports Chinese OCR plus Chinese-to-English translation. In this paper, we highlight the UI design of the system and the corresponding user-intention guided text extraction approach to achieving a compelling user experience.

[1]  Kesheng Wu,et al.  Optimizing two-pass connected-component labeling algorithms , 2009, Pattern Analysis and Applications.

[2]  Ying Zhang,et al.  Towards Automatic Sign Translation , 2001, HLT.

[3]  Avraham A. Melkman,et al.  On-Line Construction of the Convex Hull of a Simple Polyline , 1987, Inf. Process. Lett..

[4]  Daniel Vogel,et al.  Shift: a technique for operating pen-based interfaces using touch , 2007, CHI.

[5]  Matthew Turk,et al.  TranslatAR: A mobile augmented reality translator , 2011, 2011 IEEE Workshop on Applications of Computer Vision (WACV).

[6]  S. Lucas,et al.  ICDAR 2003 robust reading competitions: entries, results, and future directions , 2005, International Journal of Document Analysis and Recognition (IJDAR).

[7]  Masaaki Nagata,et al.  Portable Translator Capable of Recognizing Characters on Signboard and Menu Captured by its Built-in Camera , 2005, ACL.

[8]  Yasuhiko Watanabe,et al.  Translation camera on mobile phone , 2003, 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698).

[9]  Ismail Haritaoglu.  Scene text extraction and translation for handheld devices , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.