The Tradition of Categoricity and Prospects for Stochasticity

“Everyone knows that language is variable.” This is the bald sentence with which Sapir (1921:147) begins his chapter on language as an historical product. He goes on to emphasize how two speakers’ usage is bound to differ “in choice of words, in sentence structure, in the relative frequency with which particular forms or combinations of words are used”. I should add that much sociolinguistic and historical linguistic research has shown that the same speaker’s usage is also variable (Labov 1966, Kroch 2001:722). However, the tradition of most syntacticians has been to ignore this thing that everyone knows.1 Human languages are the prototypical example of a symbolic system. From very early on, logics and logical reasoning were invented for handling natural language understanding. Logics and formal languages have a language-like form that draws from and meshes well with natural languages. It is not immediately obvious where the continuous and quantitative aspects of syntax are. The dominant answer in syntactic theory has been “nowhere” (Chomsky 1969:57; also 1956, 1957, etc.): “It must be recognized that the notion ‘probability of a sentence’ is an entirely useless one, under any known interpretation of this term.” In the 1950s there were prospects for probabilistic methods taking hold in linguistics, in part due to the influence of the new field of Information Theory (Shannon 1948).2 Chomsky’s influential remarks had the effect of killing off interest in probabilistic methods for syntax, just as for a long time McCarthy and Hayes (1969) discouraged exploration of probabilistic methods in Artificial Intelligence. Among his arguments were that: (i) Probabilistic models wrongly mix in world knowledge (New York occurs more in text than Dayton, Ohio, but for no linguistic reason), (ii) Probabilistic models don’t model grammaticality (neither Colorless green ideas sleep furiously nor Furiously sleep ideas green colorless have previously been uttered – and hence must be estimated to have probability zero, Chomsky wrongly assumes – but the former is grammatical while the latter is not, and (iii) Use of probabilities does not meet the goal of describing the mind-internal I-language as opposed to the observed-in-the-world E-language. This chapter is not meant to be a detailed critique of Chomsky’s arguments – Abney (1996) provides a survey and a rebuttal, and Pereira (2000) has further useful discussion – but some of these concerns are still important to discuss. I