Measuring the Implementation Fidelity of Student Affairs Programs: A Critical Component of the Outcomes Assessment Cycle

Email gerstnjj@dukes.jmu.edu Abstract Implementation fidelity assessment provides a means of measuring the alignment between the planned program and the implemented program. Unfortunately, the implemented program can differ from the planned program, resulting in ambiguous inferences about the planned program’s effectiveness (i.e., it is uncertain if poor results are due to an ineffective program or poor implementation). We demonstrate how inclusion of implementation fidelity in the outcomes assessment process increases the validity of inferences about program effectiveness and, ultimately, student learning. Although our didactic discussion of implementation fidelity focuses on its importance to assessing student affairs programming, the concepts and process are applicable to academic programs as well. RESEARCH & PRACTICE IN ASSESSMENT

[1]  Susan M. Sheridan,et al.  Fidelity Measurement in Consultation: Psychometric Issues and Preliminary Examination. , 2009 .

[2]  T. Kratochwill,et al.  Treatment integrity assessment in the schools: An evaluation of the Treatment Integrity Planning Protocol. , 2009 .

[3]  J. Gee.  Opportunity to Learn: A language-based perspective on assessment , 2003 .

[4]  Emil J. Posavac,et al.  Program Evaluation: Methods and Case Studies , 1980 .

[5]  David S. Cordray,et al.  Moving From the Lab to the Field: The Role of Fidelity and Achieved Relative Intervention Strength , 2009 .

[6]  Andrew V. Dane,et al.  Program integrity in primary and early secondary prevention: are implementation effects out of control? , 1998, Clinical psychology review.

[7]  Derek D. Reed,et al.  Treatment integrity of school-based interventions with children in the journal of applied behavior analysis 1991-2005. , 2007, Journal of applied behavior analysis.

[8]  K. Arrow.  Higher education as a filter , 1973 .

[9]  L. Fogg,et al.  Measuring participation in a prevention trial with parents of young children. , 2006, Research in nursing & health.

[10]  John H. Schuh,et al.  Assessment Practice in Student Affairs: An Applications Manual , 2000 .

[11]  Morgan S. Polikoff.  Instructional Sensitivity as a Psychometric Property of Assessments , 2010 .

[12]  Andrew Booth,et al.  Implementation Science BioMed Central Debate A conceptual framework for implementation fidelity , 2007 .

[13]  J. Durlak,et al.  Implementation Matters: A Review of Research on the Influence of Implementation on Program Outcomes and the Factors Affecting Implementation , 2008, American journal of community psychology.

[14]  John H. Schuh.  Assessment Methods for Student Affairs , 2008 .

[15]  M. Mclaughlin,et al.  Implementation of Educational Innovation , 1976 .

[16]  John H. Schuh,et al.  Assessment in Student Affairs: A Guide for Practitioners. The Jossey-Bass Higher and Adult Education Series. , 1996 .

[17]  Thomas R. Kratochwill,et al.  Toward Developing a Science of Treatment Integrity: Introduction to the Special Series. , 2009 .

[18]  Deborah Bybee,et al.  Fidelity Criteria: Development, Measurement, and Validation , 2003 .

[19]  B. Resnick,et al.  Measuring Implementation Fidelity in a Community-Based Parenting Intervention , 2010, Nursing research.

[20]  Jeanne Century,et al.  Measuring Enactment of Innovations and the Factors that Affect Implementation and Sustainability: Moving Toward Common Language and Shared Conceptual Understanding , 2012, The Journal of Behavioral Health Services & Research.

[21]  Marilee J. Bresciani,et al.  Demonstrating Student Success: A Practical Guide to Outcomes-Based Assessment of Learning and Development in Student Affairs , 2010 .

[22]  J. Laux,et al.  A survey investigating school psychologists' measurement of treatment integrity in school‐based interventions and their beliefs about its importance , 2008 .

[23]  Florence D. DiGennaro Reed,et al.  Barriers to Implementing Treatment Integrity Procedures in School Psychology Research , 2012 .

[24]  M. D. Shutt,et al.  An Assessment Model as Best Practice in Student Affairs , 2012 .

[25]  A. J. Brumbaugh,et al.  American College Personnel Association , 1938 .

[26]  Lisa V. McCulley,et al.  Intervention Fidelity in Special and General Education Research Journals , 2013 .

[27]  T. Kratochwill,et al.  Treatment Integrity in Behavioral Consultation: Measurement, Promotion, and Outcomes , 2007 .

[28]  Carol L. O'Donnell.  Defining, Conceptualizing, and Measuring Fidelity of Implementation and Its Relationship to Outcomes in K–12 Curriculum Intervention Research , 2008 .

[29]  Theodore J. Christ,et al.  Supporting valid decision making: Uses and misuses of assessment data within the context of RTI , 2012 .

[30]  David S. Cordray,et al.  A Procedure for Assessing Intervention Fidelity in Experiments Testing Educational and Behavioral Interventions , 2012, The Journal of Behavioral Health Services & Research.

[31]  F. Gresham,et al.  Treatment Integrity: An Essential—But Often Forgotten—Component of School-Based Interventions , 2004 .

[32]  Christy K. Boscardin,et al.  Relationship Between Opportunity to Learn and Student Performance on English and Algebra Assessments , 2005 .