The retrospective pretest–posttest design redux: On its validity as an alternative to traditional pretest–posttest measurement

We revisit the merits of the retrospective pretest–posttest (RPP) design for repeated-measures research. The underutilized RPP method asks respondents to rate survey items twice during the same posttest measurement occasion from two specific frames of reference: “now” and “then.” Individuals first report their current attitudes or beliefs following a given intervention, and next they are prompted to think back to a specific time prior to the given intervention and rate the item again retrospectively. The design addresses many of the validity concerns that plague the traditional pretest–posttest design. Particularly when measuring noncognitive constructs, the RPP design allows participants to gauge the degree of change that they experience with greater awareness and precision than a traditional approach. We review the undesirable features of traditional designs and highlight the benefits of the retrospective approach. We offer examples from two recent, original studies and conclude with the recommendation that the RPP design be employed more broadly. We also conclude with a discussion of important directions for future examination of this design.

[1]  T. Little,et al.  From quality to outcomes: a national study of afterschool STEM programming , 2019, International Journal of STEM Education.

[2]  E. Cohen.  Self-Assessing the Benefits of Educational Tours , 2016 .

[3]  Sabrina Eberhart,et al.  Applied Missing Data Analysis , 2016 .

[4]  Todd D. Little,et al.  Using Principal Components as Auxiliary Variables in Missing Data Estimation , 2015, Multivariate behavioral research.

[5]  Each measure of patient-reported change provides useful information and is susceptible to bias: the need to combine methods to assess their relative validity. , 2009, Arthritis and rheumatism.

[6]  Cynthia A. Tananis,et al.  Measuring Change in a Short-Term Educational Program Using a Retrospective Pretest Design , 2009 .

[7]  Paul J. Taylor,et al.  Gilding the Outcome by Tarnishing the Past , 2009 .

[8]  K. Yoneda,et al.  Effect of sequential applications of topical tacrolimus and topical corticosteroids in the treatment of pediatric atopic dermatitis: an open-label pilot study. , 2009, Journal of the American Academy of Dermatology.

[9]  A. Hyde,et al.  Controlling response shift bias: the use of the retrospective pre‐test design in the evaluation of a master's programme , 2008 .

[10]  D. Moberg,et al.  Recovery High Schools: A Descriptive Study of School Programs and Students , 2008, Journal of groups in addiction & recovery.

[11]  Harris Cooper,et al.  Effect Size Substantive Interpretation Guidelines : Issues in the Interpretation of Effect Sizes , 2007 .

[12]  Todd D. Little,et al.  A Non-arbitrary Method of Identifying and Scaling Latent Variables in SEM and MACS Models , 2006 .

[13]  J. Rodgers,et al.  An empirical evaluation of the retrospective pretest: Are there advantages to looking back? , 2005 .

[14]  F. Oort.  Using structural equation modeling to detect response shifts and true change , 2005, Quality of Life Research.

[15]  Tony C. M. Lam,et al.  A Comparison of Three Retrospective Self-reporting Methods of Measuring Change in Instructional Practice , 2003 .

[16]  F. R. Rohs.  Improving the Evaluation of Leadership Programs: Control Response Shift , 2002 .

[17]  Gregory A. Davis,et al.  Using a Retrospective Pre-Post Questionnaire To Determine Program Impact. , 2002 .

[18]  Gordon W. Cheung,et al.  Evaluating Goodness-of-Fit Indexes for Testing Measurement Invariance , 2002 .

[19]  V. Willson,et al.  Methodological and statistical considerations for threats to internal validity in pediatric outcome data: response shift in self-report outcomes. , 2002, Journal of pediatric psychology.

[20]  A. Farel,et al.  Impact of an Online Analytic Skills Course , 2001, Evaluation & the health professions.

[21]  Clara C. Pratt,et al.  Measuring Program Outcomes: Using Retrospective Pretest Methodology , 2000 .

[22]  C. Schwartz,et al.  Methodological approaches for assessing response shift in longitudinal health-related quality-of-life research. , 1999, Social science & medicine.

[23]  M. Sprangers,et al.  Integrating response shift into health-related quality of life research: a theoretical model. , 1999, Social science & medicine.

[24]  M. Sprangers,et al.  Revealing response shift in longitudinal research on fatigue--the use of the thentest approach. , 1999, Acta oncologica.

[25]  M. Fekkes,et al.  Psychological responses to the needle‐free Medi‐Jector® or the multidose Disetronic® injection pen in human growth hormone therapy , 1998, Acta paediatrica.

[26]  M. Biernat,et al.  Shifting standards and stereotype-based judgments. , 1994, Journal of personality and social psychology.

[27]  F. V. van Dam,et al.  Underreporting by cancer patients: the case of response-shift. , 1991, Social science & medicine.

[28]  M. Sprangers.  RESPONSE-SHIFT BIAS IN PROGRAM EVALUATION , 1989 .

[29]  Johan Hoogstraten,et al.  Pretesting effects in retrospective pretest posttest designs. , 1989 .

[30]  M. Sprangers.  Subject bias and the retrospective pretest in retrospect , 1989 .

[31]  J. Rhodes,et al.  The Retrospective Pretest: An Alternative Approach in Evaluating Drug Prevention Programs , 1987, Journal of drug education.

[32]  Scott E. Maxwell,et al.  Methods of Analysis with Response-Shift BIAS , 1984 .

[33]  George S. Howard,et al.  Response-Shift Bias , 1980 .

[34]  George S. Howard,et al.  A Problem in Evaluating Interventions with Pre/Post Self-Reports , 1980 .

[35]  George S. Howard,et al.  The Feasibility of Informed Pretests in Attenuating Response-Shift Bias , 1979 .

[36]  G. Howard,et al.  Internal Invalidity in Studies Employing Self-Report Instruments: A Suggested Remedy. , 1979 .

[37]  George S. Howard,et al.  Response-shift bias: A source of contamination of self-report measures. , 1979 .

[38]  Scott E. Maxwell,et al.  Internal Invalidity in Pretest-Posttest Self-Report Evaluations and a Re-evaluation of Retrospective Pretests , 1979 .

[39]  L. Cronbach,et al.  How we should measure "change": Or should we? , 1970 .