Sources of Variability in Estimates of Predictive Validity

Level of Service (LS) is one of the most widely used general risk and need assessment tools in criminal justice agencies across North America. However, there is significant interstudy variability in the magnitude of the validity estimates. This study was conducted to examine possible sources of this variability. The predictive validity of LS risk and need increased with length of follow-up period and with investigator allegiance to LS. The combination of these two variables reveals consistent increases in mean predictive validity estimates from modest (in the .20s) through large (in the mid .30s) to very large (in the .40s) in samples of both male and female offenders. We hypothesized that the “allegiance effect” reflects the integrity of LS implementation and support provided by the agency for risk assessment. This is akin to the difference between “demonstration projects” and “practical” rehabilitation programming in the offender treatment research. Moreover, controls for Canadian versus non-Canadian evaluations reduced the effect of allegiance and length of follow-up to nonsignificant levels. Possible explanations for these findings include the degree of integrity in conducting risk and need assessments, the accuracy of recidivism as the criterion measure, and generalizability across international boundaries.

[1]  L. M. Laosa.  THE CULTURAL CONTEXT OF CONSTRUCT VALIDITY AND THE ETHICS OF GENERALIZABILITY , 1991 .

[2]  K. Heilbrun.  Violent offenders: Appraising and managing risk. , 1998 .

[3]  Anthony Petrosino,et al.  The impact of program developers as evaluators on criminal recidivism: Results from meta-analyses of experimental and quasi-experimental research , 2005 .

[4]  J. McGuire,et al.  Integrating Findings from Research Reviews , 2008 .

[5]  C. Hollin,et al.  The Level of Service Inventory— Revised With English Women Prisoners , 2007 .

[6]  Grant E. Coulson,et al.  Predictive Utility of the LSI for Incarcerated Female Offenders , 1996 .

[7]  Jean Folsom,et al.  The Generalizability of the LSI-R and the Cat To the Prediction of Recidivism in Female Offenders , 2007 .

[8]  C. Dowden,et al.  Managing correctional treatment for reduced recidivism: A meta‐analytic review of programme integrity , 2005 .

[9]  Christopher T. Lowenkamp,et al.  The Risk Principle in Action: What Have We Learned From 13,676 Offenders and 97 Correctional Programs? , 2006 .

[10]  Emily J. Salisbury,et al.  Achieving Accurate Pictures of Risk and Identifying Gender Responsive Needs: Two New Assessments for Women Offenders 1 , 2008 .

[11]  R. D. Hoge,et al.  Reliability and Validity Analyses of the Youth Level of Service/Case Management Inventory , 2005 .

[12]  Dan Atkinson,et al.  Risk assessment: Identifying patterns of risk in young offenders with the Youth Level of Service/Case Management Inventory , 2008 .

[13]  R. Hoge.  Youth Level of Service/Case Management Inventory , 2011 .

[14]  Duyen Luong,et al.  Applying Risk/Need Assessment to Probation Practice and its Impact on the Recidivism of Young Offenders , 2011 .

[15]  G. Harris,et al.  Allegiance or Fidelity? A Clarifying Reply , 2010 .

[16]  Michael D. Reisig,et al.  POVERTY, STATE CAPITAL, AND RECIDIVISM AMONG WOMEN OFFENDERS* , 2004 .

[17]  J. Skeem,et al.  Criminal Justice and Behavior , 2006 .

[18]  James Bonta,et al.  The Level of Service/Case Management Inventory (LS/CMI): Reliability and Validity in Female Offenders , 2009 .

[19]  Stephen C. P. Wong,et al.  The efficacy of violence prediction: a meta-analytic comparison of nine risk assessment tools. , 2010, Psychological bulletin.

[20]  D. A. Andrews,et al.  The level of service inventory – revised , 1995 .

[21]  Mark W. Lipsey,et al.  The Primary Factors that Characterize Effective Interventions with Juvenile Offenders: A Meta-Analytic Overview , 2009 .

[22]  Kristy Holtfreter,et al.  Assessing Recidivism Risk Across Female Pathways to Crime , 2006 .

[23]  D. A. Andrews,et al.  General Risk and Need, Gender Specificity, and the Recidivism of Female Offenders , 2010 .

[24]  Antonio Andrés Pueyo,et al.  THE PSYCHOLOGY OF CRIMINAL CONDUCT , 2008 .

[25]  D. Fanelli.  How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data , 2009, PloS one.

[26]  J. Mcguire.  Offender rehabilitation and treatment : effective programmes and policies to reduce re-offending , 2002 .

[27]  Christopher T. Lowenkamp,et al.  Exploring the validity of the Level of Service Inventory-Revised with Native American offenders , 2006 .

[28]  Mitchell K. Byrne,et al.  The Level of Service Inventory—Revised (LSI-R) , 2009 .

[29]  Grant T. Harris,et al.  Comparing Effect Sizes in Follow-Up Studies: ROC Area, Cohen's d, and r , 2005, Law and human behavior.

[30]  Peter Raynor,et al.  Risk and need assessment in British probation: the contribution of LSI-R , 2007 .

[31]  J. S. Wormith,et al.  The Predictive Validity of the Level of Service Inventory-Ontario Revision on General and Violent Recidivism among Various Offender Groups , 2004 .

[32]  D. A. Andrews,et al.  Manual for the Youth Level of Service Inventory , 1984 .

[33]  Shelley L. Brown,et al.  The Assessment and Treatment of Women Offenders , 2006 .

[34]  Christopher T. Lowenkamp,et al.  Predicting outcome with the Level of Service Inventory-Revised: The importance of implementation integrity , 2006 .

[35]  James Bonta,et al.  The psychology of criminal conduct, 5th ed. , 2010 .

[36]  Peter Raynor,et al.  Are the Major Risk/Need Factors Predictive of Both Female and Male Reoffending? , 2012, International journal of offender therapy and comparative criminology.

[37]  C. Schwalbe.  A Meta-Analysis of Juvenile Justice Risk Assessment Instruments , 2008 .

[38]  D. A. Andrews,et al.  The Recent Past and Near Future of Risk and/or Need Assessment , 2006 .

[39]  Francis T. Cullen,et al.  Can 14,737 women be wrong? A meta‐analysis of the LSI‐R and recidivism for female offenders* , 2009 .

[40]  W. Davidson,et al.  Predicting Recidivism in Probationers With the Youth Level of Service Case Management Inventory (YLS/CMI) , 2008 .

[41]  Kelly Hannah-Moffat.  Gridlock or mutability: Reconsidering “gender” and risk assessment† , 2009 .

[42]  Christopher T. Lowenkamp,et al.  Risk/Need Assessment, Offender Classification, and the Role of Childhood Abuse , 2001 .

[43]  James Bonta,et al.  CLASSIFICATION TO HALFWAY HOUSES: A QUASI‐EXPERIMENTAL EVALUATION * , 1990 .

[44]  J. Bonta,et al.  The Level of Service (LS) assessment of adults and older adolescents. , 2010 .

[45]  J. Stephen Wormith,et al.  Personality and crime: Knowledge destruction and construction in criminology , 1989 .

[46]  Kelly Hannah-Moffat.  V. Gendering Risk at What Cost: Negotiations of Gender and Risk in Canadian Women’s Prisons , 2004 .