Capitalizing on Variation: How Conflicting Research Findings Can Be Helpful for Policy

I first became interested in data synthesis about 10 years ago. Two circumstances triggered it. The first was reading doctoral theses. When I became a faculty member, I noticed many theses which I read had a similar organization; chapter one was a statement of a problem, chapter two was entitled "Review of the Literature," then chapters three through six presented the author's work, but nearly always independent of the literature review! So I used to write a very pleasant note to some students, telling them that their thesis looked fine but that they should be more systematic about their review and synthesis of literature. I told them that merely saying Jones (2001) found X, and Smith (2002) found Y, wasn't scientific enough. Then, in the early 1970's there was, among my colleagues, a substantial growth in 'policy research.' Part of this involved managers and coordinators of programs such as Head Start asking what, overall, was known about the effectiveness of their program. This led to my initiating a synthesis of some early educational programs. The goal was to refine the findings of many studies to a "core" outcome. It was far harder than I had imagined to do a truly systematic and rigorous analysis of existing studies. They had been done by different people in different places at different times, often with different research designs. I remember one case where it