Creating Risk-Scores in Very Imbalanced Datasets: Predicting Extremely Violent Crime among Criminal Offenders Following Release from Prison

In this chapter, we will discuss the many benefits of the Area under the Curve metric (AUC), not only as a performance measure, but also as a tool for optimizing models on very imbalanced datasets. We first introduce the measure formally and then discuss a few modeling techniques that can be used specifically for imbalanced datasets. Then we will include techniques that optimize the AUC directly. We will discuss how to choose suitable cut-points on an AUC optimized score and present a case study on predicting violent felony offenses (VFO) on a parole population.

[1]  Szymon Jaroszewicz,et al.  Efficient AUC Optimization for Classification , 2007, PKDD.

[2]  Stephen R. Garner,et al.  WEKA: The Waikato Environment for Knowledge Analysis , 1996 .

[3]  Shelley L. Brown,et al.  The Assessment and Treatment of Women Offenders , 2006 .

[4]  Stephen D. Gottfredson,et al.  Prediction: An Overview of Selected Methodological Issues , 1987, Crime and Justice.

[5]  J. Petersilia,et al.  Prisoner Reentry: Public Safety and Reintegration Challenges , 2001 .

[6]  Michael I. Jordan,et al.  Minimax Probability Machine , 2001, NIPS.

[7]  C. Stadtland,et al.  The validity of the Violence Risk Appraisal Guide (VRAG) in predicting criminal recidivism. , 2007, Criminal behaviour and mental health : CBMH.

[8]  Martin Grann,et al.  Actuarial Assessment of Risk for Violence , 2000 .

[9]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[10]  David Rowe,et al.  Up and Running , 1993 .

[11]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[12]  David T. Lykken,et al.  The Antisocial Personalities , 1995 .

[13]  D. Shanno.  Conditioning of Quasi-Newton Methods for Function Minimization , 1970 .

[14]  D. A. Andrews,et al.  The Recent Past and Near Future of Risk and/or Need Assessment , 2006 .

[15]  B. Owen,et al.  In the Mix: Struggle and Survival in a Women's Prison , 1998 .

[16]  D. A. Andrews,et al.  The level of service inventory – revised , 1995 .

[17]  Langche Zeng.  Prediction and Classification with Neural Network Models , 1999 .

[18]  Vernon L. Quinsey,et al.  Violent Recidivism of Mentally Disordered Offenders , 1993 .

[19]  W. R. Smith.  The effects of base rate and cutoff point choice on commonly used measures of association and accuracy in recidivism research , 1996 .

[20]  Sebastian Thrun,et al.  Extracting Rules from Artifical Neural Networks with Distributed Representations , 1994, NIPS.

[21]  R. Fletcher,et al.  A New Approach to Variable Metric Algorithms , 1970, Comput. J..

[22]  Thorsten Joachims,et al.  A support vector method for multivariate performance measures , 2005, ICML.

[23]  Melania Pintilie,et al.  Analysing and interpreting competing risk data , 2007, Statistics in medicine.

[24]  C. G. Broyden.  The Convergence of a Class of Double-rank Minimization Algorithms 2. The New Algorithm , 1970 .

[25]  D. Goldfarb.  A family of variable-metric methods derived by variational means , 1970 .

[26]  Michael C. Seto,et al.  Evaluating the Predictive Accuracy of Six Risk Assessment Instruments for Adult Sex Offenders , 2001 .

[27]  R. Dawes.  Judgment under uncertainty: The robust beauty of improper linear models in decision making , 1979 .

[28]  Eric Silver,et al.  A Cautionary Note on the Use of Actuarial Risk Assessment Tools for Social Control , 2002 .

[29]  Robert Gray,et al.  A Proportional Hazards Model for the Subdistribution of a Competing Risk , 1999 .

[30]  J A Swets,et al.  Measuring the accuracy of diagnostic systems. , 1988, Science.

[31]  Kathleen Daly,et al.  Women's pathways to felony court: feminist theories of lawbreaking and problems of representation , 1992 .

[32]  Larry Nazareth,et al.  A family of variable metric updates , 1977, Math. Program..

[33]  Klaus-Peter Dahle,et al.  Strengths and limitations of actuarial prediction of criminal reoffence in a German prison sample: a comparative study of LSI-R, HCR-20 and PCL-R. , 2006, International journal of law and psychiatry.

[34]  J. Maindonald.  Statistical Learning from a Regression Perspective , 2008 .

[35]  F. Harrell,et al.  Evaluating the yield of medical tests. , 1982, JAMA.

[36]  Markus Breitenbach,et al.  Towards an Explanatory Taxonomy of Adolescent Delinquents: Identifying Several Social-Psychological Profiles , 2008 .

[37]  Roxanne Lieb,et al.  Sex Offender Sentencing in Washington State , 1997 .

[38]  Rich Caruana,et al.  An empirical comparison of supervised learning algorithms , 2006, ICML.

[39]  K. Heilbrun.  Violent offenders: Appraising and managing risk. , 1998 .

[40]  Yoav Freund,et al.  The Alternating Decision Tree Learning Algorithm , 1999, ICML.

[41]  Ulf Brefeld,et al.  {AUC} maximizing support vector learning , 2005 .

[42]  A. Maden.  Rethinking Risk Assessment. The MacArthur study of Mental Disorder and Violence , 2003 .

[43]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[44]  Bernhard Schölkopf,et al.  Learning with kernels , 2001 .

[45]  M E Rice,et al.  Violent recidivism: assessing predictive validity. , 1995, Journal of consulting and clinical psychology.

[46]  Wilpen L. Gorr,et al.  Predicting criminal recidivism: A comparison of neural network models with statistical methods , 1996 .

[47]  P. Meehl,et al.  Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical–statistical controversy. , 1996 .

[48]  Richard A. BerkBrian.  Forecasting Dangerous Inmate Misconduct: An Application of Ensemble Statistical Procedures , 2006 .

[49]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[50]  Tim Brennan,et al.  Classification: An Overview of Selected Methodological Issues , 1987, Crime and Justice.

[51]  Gregory Z. Grudic,et al.  Sparse Greedy Minimax Probability Machine Classification , 2003, NIPS.

[52]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.