Strategic and Tactical Limits of Comparison Studies

A comparison study is an experiment whose primary purpose is to compare directly (regardless of experimental design) at least two different procedures for changing behavior or two or more components of such a procedure. This paper argues that, in spite of their popularity, such studies typically lead to inappropriate inferences with poor generality based on improper evidence gathered in support of the wrong question, thus wasting the limited experimental resources. The discussion considers problems concerning the functions of comparison studies, the nature of the comparisons that are attempted, the generality of their findings, and the limited role that they can play in technological research.