Linguistic Theory and Natural Language Processing

The 1960’s were a tempestuous time — a time of rebellion, a time for defining new directions. And by the end of the 1960’s the field of ‘theoretical’ Linguistics had already rebelled and defined its new direction — a path directly away from the field of Natural Language Processing (NLP) and away from Psycholinguistics. Following Chomsky’s lead, linguists showed little interest in ‘engineering applications’ like NLP.2 And once psycholinguistic studies began to contradict the transformational theories of the time (e.g. by finding that transformational complexity was often at odds with measurable real-time processing complexity), it was the psycholinguistic methods, not the linguistic theories that were dismissed. Down to the present day, most people who call themselves ‘theoretical’ linguists are in complete agreement that processing considerations have little or no role to play in the development of theories of linguistic knowledge. The ‘best theory’, it is believed, will emerge from considerations of elegance alone: the succinct expression of ‘linguistically significant generalizations’.