A User-Powered American Sign Language Dictionary

Students learning American Sign Language (ASL) have trouble searching for the meaning of unfamiliar signs. ASL signs can be differentiated by a small set of simple features including hand shape, orientation, location, and movement. In a feature-based ASL-to-English dictionary, users search for a sign by providing a query, which is a set of observed features. Because there is natural variability in the way signs are executed, and observations are error-prone, an approach other than exact matching of features is needed. We propose ASL-Search, an ASL-to-English dictionary entirely powered by its users. ASL-Search utilizes Latent Semantic Analysis (LSA) on a database of feature-based user queries to account for variability. To demonstrate ASL-Search's viability, we created ASL-Flash, a learning tool that presents online flashcards to ASL students and provides query data. Our simulations on this data serve as a proof of concept, demonstrating that our dictionary's performance improves with use and performs well for users with varied levels of ASL experience.

[1]  Luis von Ahn.  Duolingo: learn a language for free while helping to translate the web , 2013, IUI '13.

[2]  Walter S. Lasecki,et al.  Real-time captioning by groups of non-experts , 2012, UIST.

[3]  Shih-Fu Chang,et al.  Image Retrieval: Current Techniques, Promising Directions, and Open Issues , 1999, J. Vis. Commun. Image Represent..

[4]  John Le,et al.  Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution , 2010 .

[5]  John Riedl,et al.  Can people collaborate to improve the relevance of search results? , 2008, RecSys '08.

[6]  Gabriella Kazai,et al.  In Search of Quality in Crowdsourcing for Search Engine Evaluation , 2011, ECIR.

[7]  Richard A. Harshman,et al.  Indexing by Latent Semantic Analysis , 1990, J. Am. Soc. Inf. Sci..

[8]  Jaana Kekäläinen,et al.  Cumulated gain-based evaluation of IR techniques , 2002, TOIS.

[9]  Ming Ouhyoung,et al.  A real-time continuous gesture recognition system for sign language , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[10]  Tyrone Giordano.  Toward a crowdsourced model for ASL translations of Shakespeare's works , 2013 .

[11]  Cindi Trainor.  Open source, crowd source: harnessing the power of the people behind our libraries , 2009, Program.

[12]  Natalia Lusin,et al.  Enrollments in Languages other than English in United States Institutions of Higher Education, Fall 2006 , 2007 .

[13]  Alex Pentland,et al.  Real-time American Sign Language recognition from video using hidden Markov models , 1995 .

[14]  Patrick F. Reidy.  An Introduction to Latent Semantic Analysis , 2009 .

[15]  Z. Zenn Bien,et al.  A dynamic gesture recognition system for the Korean sign language (KSL) , 1996, IEEE Trans. Syst. Man Cybern. Part B.

[16]  Gabriella Kazai,et al.  Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking , 2011, SIGIR.

[17]  Nicolas Pugeault,et al.  Sign Language Recognition using Sequential Pattern Trees , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Nicolas Pugeault,et al.  Reading the signs: A video based sign dictionary , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[19]  Matthew Lease,et al.  Crowdsourcing Document Relevance Assessment with Mechanical Turk , 2010, Mturk@HLT-NAACL.

[20]  Shan Lu,et al.  Color-based hands tracking system for sign language recognition , 1998, Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition.

[21]  Thad Starner,et al.  American sign language recognition with the kinect , 2011, ICMI '11.

[22]  Filip Radlinski,et al.  Search Engines that Learn from Implicit Feedback , 2007, Computer.

[23]  Richard Bowden,et al.  Search-By-Example in Multilingual Sign Language Databases , 2011 .

[24]  Vassilis Athitsos,et al.  A System for Large Vocabulary Sign Search , 2010, ECCV Workshops.

[25]  Richard A. Tennant,et al.  The American Sign Language Handshape Dictionary , 1998 .

[26]  Narendra Ahuja,et al.  Recognizing hand gesture using motion trajectories , 1999, Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149).

[27]  Maxine Eskénazi,et al.  Clustering dictionary definitions using Amazon Mechanical Turk , 2010, Mturk@HLT-NAACL.

[28]  Jovan Popovic,et al.  Real-time hand-tracking with a color glove , 2009, SIGGRAPH '09.

[29]  Thorsten Joachims,et al.  Optimizing search engines using clickthrough data , 2002, KDD.

[30]  Tae-Seong Kim,et al.  3-D hand motion tracking and gesture recognition using a data glove , 2009, 2009 IEEE International Symposium on Industrial Electronics.

[31]  Matt Post,et al.  Constructing Parallel Corpora for Six Indian Languages via Crowdsourcing , 2012, WMT@NAACL-HLT.

[32]  Carol Peters,et al.  Report on the SIGIR 2009 workshop on the future of IR evaluation , 2009, SIGF.

[33]  Susan T. Dumais,et al.  Improving Web Search Ranking by Incorporating User Behavior Information , 2019, SIGIR Forum.

[34]  Omar Alonso,et al.  Crowdsourcing for relevance evaluation , 2008, SIGF.

[35]  Filip Radlinski,et al.  Query chains: learning to rank from implicit feedback , 2005, KDD '05.

[36]  Narendra Ahuja,et al.  Face Detection and Gesture Recognition for Human-Computer Interaction , 2001, The International Series in Video Computing.

[37]  W. Stokoe,et al.  A dictionary of American sign language on linguistic principles , 1965 .

[38]  Yong Yu,et al.  Optimizing web search using social annotations , 2007, WWW '07.

[39]  Richard Bowden,et al.  Sign Language Recognition , 2011, Visual Analysis of Humans.