Impact evaluation of job training programmes: Selection bias in multilevel models

This paper focuses on the evaluation of a job training programme composed of several different courses. The aim is to evaluate the impact of the programme for the participants with respect to non-participants, paying attention to possible differences in the effectiveness between the courses. The analysis is based on discrete data with a hierarchical structure. Multilevel modelling is the natural choice in this setting, but the results may be severely affected by selection bias. We propose a two-step procedure, which suits both the hierarchical structure and the observational nature of data. The method selects the appropriate control group, using standard results of the propensity score methodology. A suitable multilevel model is formulated, and the dependence of the results on the amount of non-random sample selection is analysed within a likelihood-based framework. As a result, rankings for comparative performances are obtained, adjusted for the amount of plausible selection bias. The procedure is illustrated with reference to a data set about a job training programme organized in Italy in the late 1990s.

[1]  J. Robins,et al.  Likelihood-based inference with singular information matrix , 2000 .

[2]  A. Aakvik,et al.  Bounding a Matching Estimator: The Case of a Norwegian Training Program , 2001 .

[3]  Petra E. Todd,et al.  Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme , 1997 .

[4]  T. Lewis,et al.  Outliers in multilevel data , 1998 .

[5]  J. Heckman,et al.  Making the Most out of Programme Evaluations and Social Experiments: Accounting for Heterogeneity in Programme Impacts , 1997 .

[6]  D. Hedeker,et al.  A random-effects ordinal regression model for multilevel analysis. , 1994, Biometrics.

[7]  Raquel Carrasco,et al.  Binary Choice with Binary Endogenous Regressors in Panel Data: Estimating the Effect of Fertility on Female Labor Participation , 1998 .

[8]  J. Robins,et al.  Likelihood-based inference with singular , 2000 .

[9]  T. Shakespeare,et al.  Observational Studies , 2003 .

[10]  Michael Lechner,et al.  Some practical issues in the evaluation of heterogeneous labour market programmes by matching methods , 2002 .

[11]  H. Goldstein,et al.  Using Examination Results as Indicators of School and College Performance , 1996 .

[12]  D. Rubin,et al.  Reducing Bias in Observational Studies Using Subclassification on the Propensity Score , 1984 .

[13]  Harvey Goldstein,et al.  League Tables and Their Limitations: Statistical Issues in Comparisons of Institutional Performance , 1996 .

[14]  J. Booth,et al.  Standard Errors of Prediction in Generalized Linear Mixed Models , 1998 .

[15]  Ian Plewis.  Modelling impact heterogeneity , 2002 .

[16]  H. Joe.  Multivariate models and dependence concepts , 1998 .

[17]  Satishs Iyengar,et al.  Multivariate Models and Dependence Concepts , 1998 .

[18]  M. Aitkin,et al.  Statistical Modelling Issues in School Effectiveness Studies , 1986 .

[19]  D. Rubin,et al.  The central role of the propensity score in observational studies for causal effects , 1983 .

[20]  D Scharfstein,et al.  Methods for Conducting Sensitivity Analysis of Trials with Potentially Nonignorable Competing Causes of Censoring , 2001, Biometrics.

[21]  D. Guilkey,et al.  Purposive Program Placement and the Estimation of Family Planning Program Effects in Tanzania , 1998 .

[22]  Ross Ihaka,et al.  Gentleman R: R: A language for data analysis and graphics , 1996 .

[23]  David Firth,et al.  Efficacy of programmes for the unemployed: discrete time modelling of duration data from a matched‐comparison study , 1999 .

[24]  J. Heckman.  The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models , 1976 .

[25]  J. Copas,et al.  Inference for Non‐random Samples , 1997 .

[26]  H. Goldstein.  Multilevel Statistical Models , 2006 .

[27]  J. Angrist,et al.  Digitized by the Internet Archive in 2011 with Funding from Estimation of Limited-dependent Variable Models with Dummy Endogenous Regressors: Simple Strategies for Empirical Practice , 2011 .

[28]  James J. Heckman,et al.  Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: the Case of Manpower Training , 1989 .

[29]  Lung-fei Lee.  Generalized Econometric Models with Selectivity , 1983 .

[30]  F. Vella,et al.  Estimating and interpreting models with endogenous treatment effects : The relationship between competing estimators of the union impact on wages , 1993 .

[31]  G. Imbens.  The Role of the Propensity Score in Estimating Dose-Response Functions , 1999 .

[32]  H. Goldstein,et al.  The Graphical Presentation of a Collection of Means , 1995 .

[33]  J. Copas,et al.  Local sensitivity approximations for selectivity bias , 2001 .

[34]  Roel Bosker,et al.  Multilevel analysis : an introduction to basic and advanced multilevel modeling , 1999 .