Overlap: What's Tested, What's Taught?.

In studying the effectiveness of different instructional practices or programs, it is possible that an outcome measure may be biased in favor of a particular practice or program because the overlap between the test and one program is greater than the overlap for the other(s) (e.g., Walker & Schaffarzick, 1974). By overlap, we mean the extent to which there is a match between the content of what is taught and the content of the test used to measure progress in performance. When overlap is not considered, one program or set of practices may look artificially better with respect to test performance than another. Although awareness of this problem is long standing, effective ways of dealing with the problem are just emerging. This paper traces some of the ways in which the overlap question has been handled and reviews empirical examples of one approach. This will allow us to examine the implications of overlap for evaluative studies, and suggest methods for evaluating and compensating for this problem. The score received on a "good" achievement test is meant to reflect the actual knowledge that an individual or group has about the domain from which that test was drawn. That is, a good test is assumed to be a sample of items from a specified domain, where domain includes both content covered and item form. The domain of instruction