Are We Asking the Right Questions? Using Cognitive Interviews to Improve Surveys in Education Research

Improving the validity and reliability of surveys is a critical part of the response to the call for improved rigor of education research, policy analysis and evaluation. Too often we create inquiry tools without validating our measures against how respondents interpret our questions, and therefore collect data of questionable quality. The purpose of this article is to demonstrate how cognitive interviews can be a useful method for improving the reliability and validity of surveys used in education research. The cognitive interviews described in this article reveal the complex thought processes in which our respondents engaged as they discussed their responses to questions concerning standards, assessments and professional development, and the unanticipated interpretations of seemingly straightforward constructs. We present a number of examples to illustrate complexities that education researchers face in trying to design large-scale surveys to answer critical educational policy questions, and the substantive insights that can be gained from the use of cognitive interviews. The use of cognitive interviews in the development of surveys has been for the most part neglected by the educational research community, and we suggest that more attention be paid to this methodology as a way of improving survey research.

[1]  Susan H. Fuhrman,et al.  The politics of curriculum and testing , 1991 .

[2]  L. Poundie Burstein,et al.  Validating National Curriculum Indicators , 1995 .

[3]  Laura M. Desimone,et al.  What Makes Professional Development Effective? Results From a National Sample of Teachers , 2001 .

[4]  Andrew C. Porter,et al.  Measuring Classroom Practice: Lessons Learned from Efforts To Describe the Enacted Curriculum--The Reform Up Close Study. CPRE Research Report Series Report #31. , 1994 .

[5]  Amanda Datnow,et al.  Teachers' Responses to Success for All: How Beliefs, Experiences, and Adaptations Shape Implementation , 2000 .

[6]  Seymour Sudman,et al.  Measurement errors in surveys , 1993 .

[7]  Jack M. Fletcher,et al.  The Role of Instruction in Learning To Read: Preventing Reading Failure in At-Risk Children. , 1998 .

[8]  Andrew C. Porter,et al.  How Do District Management and Implementation Strategies Relate to the Quality of the Professional Development That Districts Provide to Teachers , 2002 .

[9]  Milbrey W. McLaughlin,et al.  What Matters Most in Teachers' Workplace Context?. , 1992 .

[10]  P. David Pearson,et al.  Beating the Odds in Teaching All Children to Read , 1999 .

[11]  Barbara Means,et al.  When personal history repeats itself: Decomposing memories for recurring events , 1991 .

[12]  D. Cohen,et al.  Learning Policy: When State Education Reform Works , 2001 .

[13]  Christine L Borgman,et al.  Final Report to the National Science Foundation , 2002 .

[14]  Laura M. Desimone,et al.  Effects of Professional Development on Teachers’ Instruction: Results from a Three-year Longitudinal Study , 2002 .

[15]  S. Ross,et al.  The Success of Schools in Implementing Eight Restructuring Designs: A Synthesis of First‐Year Evaluation Outcomes , 1997 .

[16]  F. Mosteller,et al.  Evidence matters : randomized trials in education research , 2002 .

[17]  Thomas D. Cook,et al.  Comer's School Development Program in Prince George's County, Maryland: A Theory-Based Evaluation , 1999 .

[18]  Dennis Peters.  Toward Better Teaching , 2002 .

[19]  Stanley Presser,et al.  SURVEY PRETESTING: DO DIFFERENT METHODS PRODUCE DIFFERENT RESULTS? , 1994 .

[20]  Richard M. Ingersoll,et al.  Teacher Turnover and Teacher Shortages: An Organizational Analysis , 2001 .

[21]  Seymour Sudman,et al.  Answering Questions: Methodology for Determining Cognitive and Communicative Processes in Survey Research , 1995 .

[22]  M. Garet,et al.  In (Re)Search of Evidence-Based School Practices: Possibilities for Integrating Nationally Representative Surveys and Randomized Field Trials to Inform Educational Policy , 2002, Peabody Journal of Education.

[23]  William H. Schmidt,et al.  A splintered vision : an investigation of U.S. science and mathematics education , 1997 .

[24]  K. A. Ericsson,et al.  Verbal reports as data. , 1980 .

[25]  M. Lipsky,et al.  Street-Level Bureaucracy: Dilemmas of the Individual in Public Services. , 1982 .

[26]  Michael S. Knapp,et al.  Teaching for Meaning in High-Poverty Classrooms , 1995 .

[27]  E. Farris,et al.  Status of education reform in public elementary and secondary schools : principals' perspectives , 1998 .

[28]  D. Muncey,et al.  Preliminary Findings from a Five-Year Study of the Coalition of Essential Schools , 1993 .

[29]  Heather C. Hill,et al.  Instructional Policy and Classroom Performance: The Mathematics Reform in California , 2000, Teachers College Record: The Voice of Scholarship in Education.

[30]  Daniel P. Mayer.  Measuring Instructional Practice: Can Policymakers Trust Survey Data? , 1999 .

[31]  L. Peak.  Pursuing Excellence: A Study of U.S. Eighth-Grade Mathematics and Science Teaching, Learning, Curriculum, and Achievement in International Context. Initial Findings from the Third International Mathematics and Science Study (TIMSS). , 1996 .

[32]  Linda M. Mcneil.  Resource Review : Contradictions of School Reform: Educational Costs of Standardized Testing , 2001 .

[33]  D. Cohen.  A Revolution in One Classroom: The Case of Mrs. Oublier , 1990 .

[34]  Davina C. D. Klein,et al.  Assessing Students' Opportunity to Learn: Teacher and Student Perspectives , 2005 .

[35]  Y. Lincoln,et al.  Scientific Research in Education , 2004 .

[36]  Cathleen Stasz,et al.  Enhancing opportunity to learn measures in NCES data , 1996 .

[37]  L. Rips,et al.  The Psychology of Survey Response , 2000 .

[38]  Becky A. Smerdon,et al.  Teacher quality : a report on the preparation and qualifications of public school teachers , 1999 .

[39]  D. Cohen.  Teaching for Understanding: Challenges for Policy and Practice , 1993 .

[40]  Milbrey W. McLaughlin,et al.  Understanding Teaching in Context. , 1992 .

[41]  R. Tourangeau,et al.  Cognitive Processes Underlying Context Effects in Attitude Measurement , 1988 .

[42]  T. P. Carpenter,et al.  Using Knowledge of Children’s Mathematics Thinking in Classroom Teaching: An Experimental Study , 1989 .

[43]  G. Willis,et al.  The use of verbal report methods in the development and testing of survey questionnaires , 1991 .

[44]  T. Cook.  Sciencephobia: Why education researchers reject randomized experiments , 2001 .

[45]  Paul Cobb,et al.  Assessment of a problem-centered second-grade mathematics project. , 1991 .

[46]  Timothy D. Wilson,et al.  Telling more than we can know: Verbal reports on mental processes. , 1977 .

[47]  Toward Better Teaching: Professional Development in 1993-94. , 1999 .

[48]  John B. Willett,et al.  By Design: Planning Research on Higher Education , 1990 .

[49]  Philip W. Jackson,et al.  The practice of teaching , 1986 .