Testing Conceptual Frameworks of Nonexperimental Program Evaluation Designs Using Structural Equation Modeling

A theory-driven approach to evaluation (TDE) emphasizes the development and empirical testing of conceptual models to understand the processes and mechanisms through which programs achieve their intended goals. However, most reported applications of TDE are limited to large-scale experimental/quasi-experimental program evaluation designs. Very few (limited) examples of the relevance of TDE to nonexperimental program evaluation designs exist in literature. Using the method of structural equation modeling to analyze data from the Interns for Indiana (IfI) program, this study demonstrates how evaluation practitioners can test logical and sequential relationships among tiers of outcomes of nonexperimental programs, especially programs with limited datasets. The study also describes how the empirical feedback can be used to understand program dynamics and improve program implementation and evaluation.

[1]  Steven Yale Sussman,et al.  Handbook of Program Development for Health Behavior Research and Practice , 2001 .

[2]  Bringing It All Together: Modeling Intervention Processes Using Structural Equation Modeling , 1999 .

[3]  D. Kolb,et al.  Learning Styles and Learning Spaces: Enhancing Experiential Learning in Higher Education , 2005 .

[4]  J. Hektner.  When Moving Up Implies Moving Out: Rural Adolescent Conflict in the Transition to Adulthood. , 1995 .

[5]  M. Greenberg,et al.  Executive functions and school readiness intervention: Impact, moderation, and mediation in the Head Start REDI program , 2008, Development and Psychopathology.

[6]  Tapabrata Maiti,et al.  Principles and Practice of Structural Equation Modeling (2nd ed.) , 2006 .

[7]  Melvin M. Mark,et al.  From program theory to tests of program theory , 1990 .

[8]  Carol H. Weiss,et al.  Nothing as Practical as Good Theory : Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families , 2011 .

[9]  M. Sobel.  Some New Results on Indirect Effects and Their Standard Errors in Covariance Structure Models , 1986 .

[10]  Edward H Livingston,et al.  The mean and standard deviation: what does it all mean? , 2004, The Journal of surgical research.

[11]  Stewart I. Donaldson,et al.  Mediator and Moderator Analysis in Program Development , 2001 .

[12]  Tanya N. Beran,et al.  Structural equation modeling in medical research: a primer , 2010, BMC Research Notes.

[13]  Huey-tsyh Chen.  Theory-driven evaluations , 1990 .

[14]  D. Kolb.  Experiential Learning: Experience as the Source of Learning and Development , 1983 .

[15]  K. Lewin.  Field theory in social science , 1951 .

[16]  Jacob Cohen,et al.  A power primer. , 1992, Psychological bulletin.

[17]  Leonard J. Huggins,et al.  Explaining the “Brain Drain” from Older Industrial Cities: The Pittsburgh Region , 2003 .

[18]  Stewart I. Donaldson,et al.  Theory-Driven Evaluation of the Work and Health Initiative: A Focus on Winning New Jobs , 2002 .

[19]  Kristopher J Preacher,et al.  Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models , 2008, Behavior research methods.

[20]  C. H. Camargo,et al.  Species distribution and susceptibility profile of Candida species in a Brazilian public tertiary hospital , 2010, BMC Research Notes.

[21]  Scott R. Rosas,et al.  Concept Mapping as a Technique for Program Theory Development , 2005 .

[22]  Lee Sechrest,et al.  Putting Program Theory into Operation , 1999 .

[23]  Rex B. Kline,et al.  Principles and Practice of Structural Equation Modeling , 1998 .

[24]  William B. Hansen,et al.  The law of maximum expected potential effect: constraints placed on program effectiveness by mediator relationships , 1996 .

[25]  M. Sobel.  Asymptotic Confidence Intervals for Indirect Effects in Structural Equation Models , 1982 .

[26]  M. Hennessy,et al.  Bringing It All Together: Modeling Intervention Processes Using Structural Equation Modeling , 1999 .

[27]  Nick L. Smith,et al.  Using path analysis to develop and evaluate program theory and impact , 1990 .

[28]  Carol Hirschon Weiss,et al.  Theory-Based Evaluation in Practice , 2000, Evaluation review.

[29]  Stewart I. Donaldson,et al.  Program Theory-Driven Evaluation Science: Strategies and Applications , 2007 .

[30]  James A. Graham,et al.  The Use of Multiple Evaluation Approaches in Program Evaluation , 2005 .

[31]  Amanda J. Fairchild,et al.  Evaluating mediation and moderation effects in school psychology: a presentation of methods and review of current practice. , 2010, Journal of school psychology.