Comparison of Dimension Reduction Methods for Automated Essay Grading

Automatic Essay Assessor (AEA) is a system that utilizes information retrieval techniques such as Latent Semantic Analysis (LSA), Probabilistic Latent Semantic Analysis (PLSA), and Latent Dirichlet Allocation (LDA) for automatic essay grading. The system uses learning materials and relatively few teacher-graded essays for calibrating the scoring mechanism before grading. We performed a series of experiments using LSA, PLSA and LDA for document comparisons in AEA. In addition to comparing the methods on a theoretical level, we compared the applicability of LSA, PLSA, and LDA to essay grading with empirical data. The results show that the use of learning materials as training data for the grading model outperforms the k-NN-based grading methods. In addition to this, we found that using LSA yielded slightly more accurate grading than PLSA and LDA. We also found that the division of the learning materials in the training data is crucial. It is better to divide learning materials into sentences than paragraphs.

[1]  Richard A. Harshman,et al.  Indexing by Latent Semantic Analysis , 1990, J. Am. Soc. Inf. Sci..

[2]  Peter W. Foltz,et al.  Supporting Content-Based Feedback in On-Line Writing Evaluation with LSA , 2000, Interact. Learn. Environ..

[3]  Leah S. Larkey,et al.  Automatic essay grading using text categorization techniques , 1998, SIGIR '98.

[4]  E. Sutinen,et al.  Automatic assessment of the content of essays based on course materials , 2004, ITRE 2004. 2nd International Conference Information Technology: Research and Education.

[5]  Heikki Mannila,et al.  Random projection in dimensionality reduction: applications to image and text data , 2001, KDD '01.

[6]  Lawrence M. Rudner,et al.  Automated Essay Scoring Using Bayes' Theorem , 2002 .

[7]  Ata Kabán,et al.  On an equivalence between PLSI and LDA , 2003, SIGIR.

[8]  Susan T. Dumais,et al.  Personalized information delivery: an analysis of information filtering methods , 1992, CACM.

[9]  Craig Boutilier,et al.  Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence , 2000 .

[10]  Eileen Kintsch,et al.  Summary Street: Interactive Computer Support for Writing , 2004 .

[11]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[12]  P. Praks,et al.  Latent Semantic Indexing for Image Retrieval Systems , 2003 .

[13]  Byoung-Tak Zhang,et al.  An Empirical Study on Dimensionality Optimization in Text Mining for Linguistic Knowledge Acquisition , 2003, PAKDD.

[14]  Arthur C. Graesser,et al.  Approximate Natural Language Understanding for an Intelligent Tutor , 1999, FLAIRS Conference.

[15]  Pierre Baldi,et al.  Modeling the Internet and the Web: Probabilistic Methods and Algorithms. By Pierre Baldi, Paolo Frasconi, Padhraic Smith, John Wiley and Sons Ltd., West Sussex, England, 2003. 285 pp ISBN 0 470 84906 1 , 2006, Inf. Process. Manag..

[16]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[17]  Tom Minka,et al.  Expectation-Propogation for the Generative Aspect Model , 2002, UAI.

[18]  Naftali Tishby,et al.  Sufficient Dimensionality Reduction , 2003, J. Mach. Learn. Res..

[19]  Patrick F. Reidy.  An Introduction to Latent Semantic Analysis , 2009 .

[20]  Peter W. Foltz,et al.  The intelligent essay assessor: Applications to educational technology , 1999 .

[21]  Luo Si,et al.  Adjusting Mixture Weights of Gaussian Mixture Model via Regularized Probabilistic Latent Semantic Analysis , 2005, PAKDD.

[22]  Bob Rehder,et al.  How Well Can Passage Meaning be Derived without Using Word Order? A Comparison of Latent Semantic Analysis and Humans , 1997 .

[23]  Erkki Sutinen,et al.  Automatic Essay Grading with Probabilistic Latent Semantic Analysis , 2005 .

[24]  Randy M. Kaplan,et al.  SCORING ESSAYS AUTOMATICALLY USING SURFACE FEATURES , 1998 .

[25]  William Wresch,et al.  The Imminence of Grading Essays by Computer-25 Years Later , 1993 .

[26]  Fabrizio Sebastiani,et al.  Machine learning in automated text categorization , 2001, CSUR.

[27]  Haym Hirsh,et al.  Using LSI for text classification in the presence of background text , 2001, CIKM '01.

[28]  Byoung-Tak Zhang,et al.  A Comparative Evaluation of Data-driven Models in Translation Selection of Machine Translation , 2002, COLING.

[29]  Erkki Sutinen,et al.  Applying Latent Dirichlet Allocation to Automatic Essay Grading , 2006, FinTAL.

[30]  Peter W. Foltz,et al.  An introduction to latent semantic analysis , 1998 .

[31]  Thorsten Brants,et al.  Test Data Likelihood for PLSA Models , 2005, Information Retrieval.

[32]  Thomas Hofmann,et al.  Unsupervised Learning by Probabilistic Latent Semantic Analysis , 2004, Machine Learning.

[33]  Martin Chodorow,et al.  Automated Scoring Using A Hybrid Feature Identification Technique , 1998, ACL.

[34]  E. B. Page,et al.  The Computer Moves into Essay Grading: Updating the Ancient Test. , 1995 .