Usability testing, initial implementation, and formative evaluation of an evidence-based intervention: lessons from a demonstration project to reduce long-term foster care.

The field of child welfare faces an undersupply of evidence-based interventions to address long-term foster care. The Permanency Innovations Initiative is a five-year federal demonstration project intended to generate evidence to reduce long stays in foster care for those youth who encounter the most substantial barriers to permanency. This article describes a systematic and staged approach to implementation and evaluation of a PII project that included usability testing as one of its key activities. Usability testing is an industry-derived practice which analyzes early implementation processes and evaluation procedures before they are finalized. This article describes the iterative selection, testing, and analysis of nine usability metrics that were designed to assess three important constructs of the project's initial implementation and evaluation: intervening early, obtaining consent, and engaging parents. Results showed that seven of nine metrics met a predetermined target. This study demonstrates how findings from usability testing influenced the initial implementation and formative evaluation of an evidence-supported intervention. Implications are discussed for usability testing as a quality improvement cycle that may contribute to better operationalized interventions and more reliable, valid, and replicable evidence.

[1]  Tamara L Fuller,et al.  Engagement in child protective services: Parent perceptions of worker skills , 2013 .

[2]  Leah Bartley,et al.  Active Implementation Frameworks for Program Success: How to Use Implementation Science to Improve Outcomes for Children. , 2012 .

[3]  Gary C. Dumbrill.  Parental experience of child protection intervention: a qualitative study. , 2006, Child abuse & neglect.

[4]  N. Harris.  Assessment: when does it help and when does it hinder? Parents' experiences of the assessment process , 2012 .

[5]  P. Dale.  ‘Like a fish in a bowl’: parents' perceptions of child protection services , 2004 .

[6]  W. Edwards Deming,et al.  Out of the Crisis , 1982 .

[7]  David L. Solomon,et al.  Confirmative Evaluation: A New Paradigm for Continuous Improvement. , 1997 .

[8]  M. Courtney.  Factors Associated with the Reunification of Foster Children with Their Families , 1994, Social Service Review.

[9]  Thomas J. Cook,et al.  Avoiding type III error in program evaluation: Results from a field experiment , 1980 .

[10]  Janette K. Klingner,et al.  Barriers and Facilitators in Scaling up Research-Based Practices , 2003 .

[11]  Cecilia Ayón,et al.  Learning How to Dance with the Public Child Welfare System: Mexican Parents' Efforts to Exercise Their Voice , 2010 .

[12]  Karen A. Blase,et al.  Statewide Implementation of Evidence-Based Programs , 2013 .

[13]  Dean L. Fixsen,et al.  Core Implementation Components , 2009 .

[14]  R. Barth,et al.  Effects of age and race on the odds of adoption versus remaining in long-term out-of-home care. , 1997, Child welfare.

[15]  K. Katz,et al.  Leaving foster care--the influence of child and case characteristics on foster care exit rates , 2006 .

[16]  Becci A. Akin,et al.  Defining a target population at high risk of long-term foster care: barriers to permanency for families of children with serious emotional disturbances. , 2012, Child welfare.

[17]  N. Carr,et al.  'Like walking on eggshells': Service user views and expectations of the child protection system , 2011 .

[18]  R. Resar,et al.  Basics of quality improvement in health care. , 2007, Mayo Clinic proceedings.

[19]  H. Mintrop,et al.  Design Cooperation: Strengthening the Link between Organizational and Instructional Change in Schools , 2001 .

[20]  Mark E. Courtney,et al.  Comparing the timing of exits from substitute care , 1996 .

[21]  K. Wells,et al.  Research on Timing of Foster Care Outcomes: One Methodological Problem and Approaches to Its Solution , 2003, Social Service Review.

[22]  J. Hawkins.  The ethics of Zelen consent , 2004, Journal of thrombosis and haemostasis : JTH.

[23]  G. Patterson,et al.  Parent Management Training—Oregon Model: An intervention for antisocial behavior in children and adolescents. , 2010 .

[24]  Becci A. Akin,et al.  Selecting an EBP to Reduce Long-Term Foster Care: Lessons from a University–Child Welfare Agency Partnership , 2014, Journal of evidence-based social work.

[25]  Mark W. Fraser,et al.  Intervention Research: Developing Social Programs , 2009 .

[26]  D. Olds,et al.  Taking preventive intervention to scale: The nurse-family partnership * , 2003 .

[27]  Mark F Testa,et al.  Insuring the Integrity and Validity of Social Work Interventions: The Case of the Subsidized Guardianship Waiver Experiments , 2014, Journal of evidence-based social work.

[28]  W. D. Diorio.  Parental Perceptions of the Authority of Public Child Welfare Caseworkers , 1992 .

[29]  B. Joyce,et al.  Student Achievement Through Staff Development , 1988 .

[30]  Jakob Nielsen,et al.  A mathematical model of the finding of usability problems , 1993, INTERCHI.

[31]  K. Hoagwood,et al.  Engaging parents in child welfare services: bridging family needs and child welfare mandates. , 2009, Child welfare.

[32]  Becci A. Akin.  Predictors of foster care exits to permanency: A competing risks analysis of reunification, guardianship, and adoption , 2011 .

[33]  M Zelen,et al.  Randomized consent designs for clinical trials: an update. , 1990, Statistics in medicine.

[34]  M. Zelen.  A new design for randomized clinical trials. , 1979, The New England journal of medicine.

[35]  W. Richardson,et al.  The well-built clinical question: a key to evidence-based decisions. , 1995, ACP journal club.

[36]  K. Klein,et al.  The Challenge of Innovation Implementation , 1996 .

[37]  Karen A. Blase,et al.  Implementation Research: A Synthesis of the , 2005 .