The design and analysis of language experiments in communication

Abstract

The traditional design for the study of language variables is a controlled message design in which one kernel message is manipulated to produce message variations that represent the language variable contrasts. Thus message content is the same across treatment conditions and hence crossed with the treatment factors. Jackson and Jacobs (1983) criticized this design on three grounds. They argued that messages should be natural rather than contrived, that messages should be nested within treatment conditions rather than crossed with treatment factors, and that each study should use as many messages as possible. Their paper has been used by reviewers as the basis for rejecting studies done using the traditional design. This paper presents a detailed analysis of the arguments presented in Jackson and Jacobs (1983) as well as subsequent papers by Jackson (1984) and Jackson and Jacobs (1987). The arguments are shown to contain both methodological and substantive errors. When the arguments are corrected, it is shown that nested designs and multiple messages should be used only when absolutely necessary. It is the traditional controlled message, crossed design and not the naturalistic message, nested design which is robust; robust against not only incidental but fundamental confounding, robust against message by treatment by message interactions, and relatively robust against Type II error. Taken together with meta‐analysis, the traditional single message, crossed design can detect and identify message by treatment interactions. On the other hand, the multiple message design makes it impossible to identify such interactions.