AUTOMATIC ASSESSMENT OF VOCABULARY USAGE WITHOUT NEGATIVE EVIDENCE

This report describes the implementation and evaluation of an automated statistical method for assessing an examinee's use of vocabulary words in constructed responses. The grammatical error-detection system, ALEK (Assessing Lexical Knowledge), infers negative evidence from the low frequency or absence of constructions in 30 million words of well-formed, copy-edited text from North American newspapers. ALEK detects two types of errors: those that violate basic principles of English syntax (e.g., agreement errors as in a desks) and those that show a lack of information about a specific word (e.g., treating a mass noun as a count noun in a pollution). The system evaluated word usage in essay-length responses to Test of English as a Foreign Language (TOEFL®) prompts. ALEK was developed using three words and was evaluated on an additional 20 words that appeared frequently in TOEFL essays and in a university word list. System accuracy was evaluated to investigate its potential for scoring performance-based measures of communicative competence. It performed with about 80% precision and 20% recall. False positives (correct usages that ALEK identified as errors) and misses (usage errors that were not recognized by ALEK) were analyzed, and methods for improving system performance were outlined.

[1]  Yaacov Choueka,et al.  Disambiguation by short contexts , 1985, Comput. Humanit..

[2]  Jacob Cohen,et al.  Applied multiple regression/correlation analysis for the behavioral sciences , 1979 .

[3]  W. Nelson Francis,et al.  FREQUENCY ANALYSIS OF ENGLISH USAGE: LEXICON AND GRAMMAR , 1983 .

[4]  Adam Kilgarriff,et al.  Introduction to the Special Issue on SENSEVAL , 2000, Comput. Humanit..

[5]  L. Dekang,et al.  Extracting collocations from text corpora , 1998 .

[6]  Srinivas Bangalore,et al.  The Institute For Research In Cognitive Science Disambiguation of Super Parts of Speech ( or Supertags ) : Almost Parsing by Aravind , 1995 .

[7]  Kathleen F. McCoy,et al.  Recognizing Syntactic Errors in the Writing of Second Language Learners , 1998, ACL.

[8]  D Nicholls,et al.  The Cambridge Learner Corpus-Error coding and analysis , 1999 .

[9]  David Yarowsky,et al.  One Sense per Collocation , 1993, HLT.

[10]  Claudia Leacock,et al.  Polysemy: Theoretical and Computational Approaches , 2000 .

[11]  Phillip Rowles.  Teaching and Learning Vocabulary , 2003 .

[12]  Abraham Kaplan,et al.  An experimental study of ambiguity and context , 1955, Mech. Transl. Comput. Linguistics.

[13]  Martin Chodorow,et al.  Automated Essay Scoring for Nonnative English Speakers , 1999 .

[14]  George A. Miller,et al.  A Topical/Local Classifier for Word Sense Identification , 2000, Comput. Humanit..

[15]  George A. Miller,et al.  Using Corpus Statistics and WordNet Relations for Sense Identification , 1998, CL.

[16]  Geoffrey Washburn,et al.  Checking grammatical mistakes for English-as-a-second-language (ESL) students , 1997 .

[17]  G. Miller,et al.  How children learn words. , 1987, Scientific American.