Cognitive Processing of Self-Report Items in Educational Research: Do They Think What We Mean?

Techniques emerging from the considerable research on cognitive aspects of survey methodology include various forms of probing and cognitive interviewing. These techniques are used to examine whether respondents' interpretations of self-report items are consistent with researchers' assumptions and intended meanings given the constructs the items are designed to measure. However, although informal procedures are common, such developments have not been systematically applied in educational research. We describe how information derived from the systematic application of cognitive pretesting can contribute to determining the validity—designated cognitive validity—of self-report items. Examples are presented from prominent motivation-related instruments that assess real-world instructional practices, mastery classroom goal structure, and student self-efficacy. The implications and pragmatics of adopting this approach are discussed.

[1]  L. Bartoshuk,et al.  On Self Reports , 2002 .

[2]  Samuel Messick,et al.  Standards-Based Score Interpretation: Establishing Valid Grounds for Valid Inferences. Research Report. , 1994 .

[3]  Jacquelynne S. Eccles,et al.  Motivation to succeed. , 1998 .

[4]  Susy Macqueen,et al.  Validity , 1973, Just Algorithms.

[5]  Duane F. Alwin,et al.  Research on Survey Quality , 1991 .

[6]  T. Wilson.  Strangers to Ourselves: Discovering the Adaptive Unconscious , 2002 .

[7]  Eric M. Anderman,et al.  Classroom goal structure, student motivation, and academic achievement. , 2006, Annual review of psychology.

[8]  P. Pintrich,et al.  Personal Epistemology The Psychology of Beliefs About Knowledge and Knowing , 2002 .

[9]  P. Pintrich,et al.  Motivational and self-regulated learning components of classroom academic performance. , 1990 .

[10]  Roger Tourangeau,et al.  Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. , 1987 .

[11]  J. Jobe,et al.  Cognitive research improves questionnaires. , 1989, American journal of public health.

[12]  D. Singer,et al.  A Piaget Primer: How a Child Thinks , 1978 .

[13]  C. R. Snyder,et al.  Social Cognitive Psychology: History and Current Domains , 1997 .

[14]  T. Urdan.  Using Multiple Methods to Assess Students' Perceptions of Classroom Goal Structures , 2004 .

[15]  H. Patrick.  RE-EXAMINING CLASSROOM MASTERY GOAL STRUCTURE , 2004 .

[16]  P. Pintrich.  A Motivational Science Perspective on the Role of Student Motivation in Learning and Teaching Contexts. , 2003 .

[17]  Michael T. Kane,et al.  An argument-based approach to validity. , 1992 .

[18]  W. Foddy.  An Empirical Evaluation of In-Depth Probes Used to Pretest Survey Questions , 1998 .

[19]  Michael J. Middleton,et al.  Achievement goals and goal structures. , 2002 .

[20]  M. Kosinski,et al.  Calibration of an item pool for assessing the burden of headaches: An application of item response theory to the Headache Impact Test (HIT™) , 2003, Quality of Life Research.

[21]  Richard F. Catalano,et al.  Measuring risk and protective factors for substance use, delinquency, and other adolescent problem behaviors. The Communities That Care Youth Survey. , 2002 .

[22]  Paul R. Pintrich,et al.  Motivating students, improving schools : the legacy of Carol Midgley , 2004 .

[23]  Alija Kulenović,et al.  Standards for Educational and Psychological Testing , 1999 .

[24]  J. Fleiss,et al.  Intraclass correlations: uses in assessing rater reliability. , 1979, Psychological bulletin.

[25]  Susan Bobbitt Nolen,et al.  Learning environment, motivation, and achievement in high school science , 2003 .

[26]  N. Schwarz,et al.  Thinking About Answers: The Application of Cognitive Processes to Survey Methodology , 1995, Quality of Life Research.

[27]  Michael E. Woolley,et al.  The Development and Evaluation of Procedures to Assess Child Self-Report Item Validity Educational and Psychological Measurement , 2006, Educational and psychological measurement.

[28]  M. Kane.  Current Concerns in Validity Theory , 2001 .

[29]  E. McColl.  Cognitive Interviewing. A Tool for Improving Questionnaire Design , 2006, Quality of Life Research.

[30]  Mimi Bong,et al.  Academic Self-Concept and Self-Efficacy: How Different Are They Really? , 2003 .

[31]  Laura M. Desimone,et al.  Are We Asking the Right Questions? Using Cognitive Interviews to Improve Surveys in Education Research , 2004 .

[32]  최영한,et al.  미국 NCTM의 Principles and Standards for School Mathematics에 나타난 수학과 교수,학습의 이론 , 2002 .

[33]  A. Elliot.  A Conceptual History of the Achievement Goal Construct. , 2005 .

[34]  N. Schwarz.  Self-reports: How the questions shape the answers. , 1999 .

[35]  Marc H. Bornstein,et al.  Social, emotional, and personality development , 1988 .

[36]  A. Su,et al.  The National Council of Teachers of Mathematics , 1932, The Mathematical Gazette.

[37]  K. Race,et al.  Assessing Student Perceptions of Classroom Methods and Activities in the Context of an Outcomes-Based Evaluation , 2000 .

[38]  C. Hirsch.  Curriculum and Evaluation Standards for School Mathematics , 1988 .

[39]  Geoffrey B. Saxe,et al.  The Value (and Convergence) of Practices Suggested by Motivation Research and Promoted by Mathematics Education Reformers. , 1998 .

[40]  P. Alexander,et al.  A Motivated Exploration of Motivation Terminology. , 2000, Contemporary educational psychology.

[41]  C. R. Snyder,et al.  Social Cognitive Psychology , 1997 .

[42]  Tim Urdan,et al.  The goals behind performance goals , 2006 .

[43]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[44]  Shameem Nyla.  NATIONAL COUNCIL ON MEASUREMENT IN EDUCATION , 2004 .

[45]  A. Ryan,et al.  The Classroom Social Environment and Changes in Adolescents’ Motivation and Engagement During Middle School , 2001 .

[46]  Carole A. Ames.  Classrooms: Goals, structures, and student motivation. , 1992 .

[47]  M. Larsen,et al.  The Psychology of Survey Response , 2002 .

[48]  Michael E. Woolley,et al.  Cognitive Pretesting and the Developmental Validity of Child Self-Report Instruments: Theory and Applications , 2004, Research on social work practice.

[49]  R. Tourangeau,et al.  Cognitive Processes Underlying Context Effects in Attitude Measurement , 1988 .

[50]  James G. Greeno,et al.  Theoretical and Practical Advances Through Research on Learning , 2006 .

[51]  Emily R. Hoole,et al.  What Are You Thinking? Postsecondary Student Think-Alouds of Scientific and Quantitative Reasoning Items , 2006, The Journal of General Education.

[52]  W. Camara,et al.  A Historical Perspective and Current Views on the Standards for Educational and Psychological Testing , 2006 .

[53]  Theresa J. DeMaio,et al.  Cognitive interviewing techniques: In the lab and in the field. , 1996 .

[54]  Roger Tourangeau,et al.  Cognition and survey research , 2000 .

[55]  Roger Tourangeau,et al.  Cognitive Aspects of Survey Measurement and Mismeasurement , 2003 .

[56]  Philip H. Winne,et al.  Measuring Self-Regulated Learning , 2000 .

[57]  A. Bandura.  Self-Efficacy: The Exercise of Control , 1997, Journal of Cognitive Psychotherapy.

[58]  S. Messick.  Validity of Psychological Assessment: Validation of Inferences from Persons' Responses and Performances as Scientific Inquiry into Score Meaning. Research Report RR-94-45. , 1994 .

[59]  R. Shavelson,et al.  On the Validity of Cognitive Interpretations of Scores From Alternative Concept-Mapping Techniques , 2001 .

[60]  James W. Pellegrino,et al.  Knowing What Students Know. , 2003 .

[61]  R. Hastie.  Information Processing Theory for the Survey Researcher , 1987 .