The design and analysis of language experiments in communication

The traditional design for the study of language variables is a controlled message design in which one kernel message is manipulated to produce message variations that represent the language variable contrasts. Thus message content is the same across treatment conditions and hence crossed with the treatment factors. Jackson and Jacobs (1983) criticized this design on three grounds. They argued that messages should be natural rather than contrived, that messages should be nested within treatment conditions rather than crossed with treatment factors, and that each study should use as many messages as possible. Their paper has been used by reviewers as the basis for rejecting studies done using the traditional design. This paper presents a detailed analysis of the arguments presented in Jackson and Jacobs (1983) as well as subsequent papers by Jackson (1984) and Jackson and Jacobs (1987). The arguments are shown to contain both methodological and substantive errors. When the arguments are corrected, it is sh...