How Well Can Passage Meaning be Derived without Using Word Order? A Comparison of Latent Semantic Analysis and Humans

How much of the meaning of a naturally occurring English passage is derivable from its combination of words without considering their order? An exploratory approach to this question was provided by asking humans to judge the quality and quantity of knowledge conveyed by short student essays on scientific topics and comparing the interrater reliability and predictive accuracy of their estimates with the performance of a corpus-based statistical model that takes no account of word order within an essay. There was surprisingly little difference between the human judges and the model. In the studies reported here, experts were asked to read short student essays about scientific topics with the goal of determining how much knowledge was accurately reflected in a given essay. We measured the readers’ success by how well their ratings agreed with each other and how well they predicted scores on an objective test on the same subject. All current accounts of human discourse understanding