A systems approach to policy evaluation

There is growing interest in evaluating policy implementation in ways that grapple with the complexity of the process. This article offers an example of using systems methodology to explore how the child protection policy in child contact centres has functioned in practice. Rather than just asking the traditional evaluation question ‘is it working?’ this study sought to understand how the policy was working and how it was interpreted as it interacted with other systems, producing conflicts, local variation and emergent effects. It illustrates how the systems concepts of ‘emergence’, ‘local rationality’, ‘socio-technical systems’ and ‘feedback for learning’ can contribute new knowledge and understanding to a complex policy evaluation problem.

[1]  Richard Hummelbrunner Systems thinking and evaluation , 2011 .

[2]  M. Reynolds Equity-focused developmental evaluation using critical systems thinking , 2014 .

[3]  E. Munro Learning to Reduce Risk in Child Protection , 2010 .

[4]  Anjum Naweed,et al.  Joint cognitive systems: Patterns in cognitive systems engineering , 2008 .

[5]  David Woods,et al.  Behind human error : cognitive systems, computers, and hindsight : state-of-the-art report , 1994 .

[6]  J. E. Groves,et al.  Made in America: Science, Technology and American Modernist Poets , 1989 .

[7]  Nancy Cartwright,et al.  Evidence-Based Policy: A Practical Guide to Doing It Better , 2012 .

[8]  David Byrne,et al.  Evaluating complex social interventions in a complex world , 2013 .

[9]  Edward J. Waller Beyond Engineering: How Society Shapes Technology , 2004 .

[10]  Sidney Dekker,et al.  Ten Questions About Human Error : A New View of Human Factors and System Safety , 2004 .

[11]  H. Anheier What Kind of Nonprofit Sector, What Kind of Society? , 2009 .

[12]  Robert R. Hoffman,et al.  Studying Cognitive Systems in Context: Preface to the Special Section , 2000, Hum. Factors.

[13]  C. Hood,et al.  The Government of Risk: Understanding Risk Regulation Regimes , 2001 .

[14]  Jesper Simonsen,et al.  Soft Systems Methodology , 2020, Systems Approaches to Making Change: A Practical Guide.

[15]  C. Hood A PUBLIC MANAGEMENT FOR ALL SEASONS , 1991 .

[16]  Christopher Pollitt,et al.  Performance or Compliance , 1999 .

[17]  Sidney W. A. Dekker,et al.  Doctors Are More Dangerous Than Gun Owners: A Rejoinder to Error Counting , 2007, Hum. Factors.

[18]  Carlos Alberto Sampaio de Freitas,et al.  Performance or compliance? Performance audit and public management in five countries , 2001 .

[19]  James T. Reason,et al.  Managing the risks of organizational accidents , 1997 .

[20]  David C. Lane,et al.  Blending systems thinking approaches for organisational analysis: Reviewing child protection in England , 2016, Eur. J. Oper. Res..

[21]  C. Willow Participation in Practice: Children and Young People as Partners in Change , 2002 .

[22]  S. Dekker,et al.  The complexity of failure: Implications of complexity theory for safety investigations , 2011 .

[23]  Nancy G. Leveson,et al.  A new accident model for engineering safer systems , 2004 .

[24]  B. Sckell,et al.  Just Culture: Balancing Safety and Accountability , 2009 .

[25]  C. Hood,et al.  Accident and Design: Contemporary Debates on Risk Management , 1996 .

[26]  Robert L. Wears,et al.  Resilience Engineering: Concepts and Precepts , 2006, Quality and Safety in Health Care.

[27]  David Woods,et al.  A tale of two stories: contrasting views of patient safety , 1998 .

[28]  L. Salamon Of Market Failure, Voluntary Failure, and Third-Party Government: Toward a Theory of Government-Nonprofit Relations in the Modern Welfare State , 1987 .

[29]  Andrea Coulson,et al.  Organized uncertainty: designing a world of risk management , 2008 .

[30]  E. Hollnagel Handbook of Cognitive Task Design , 2009 .

[31]  Sidney W A Dekker,et al.  Reconstructing human contributions to accidents: the new view on error and performance. , 2002, Journal of safety research.

[32]  S. Dekker Failure to adapt or adaptations that fail: contrasting models on procedures and safety. , 2003, Applied ergonomics.

[33]  R. Helmreich On error management: lessons from aviation , 2000, BMJ : British Medical Journal.

[34]  R. Kramer Voluntary Agencies and the Contract Culture: "Dream or Nightmare?" , 1994, Social Service Review.

[35]  Sidney Dekker,et al.  Why we need new accident models , 2004 .

[36]  Marian Barnes,et al.  Evidence, Understanding and Complexity , 2003 .

[37]  Sophie Keller,et al.  Strategic Management And Organisational Dynamics The Challenge Of Complexity To Ways Of Thinking About Organisations , 2016 .

[38]  L. Caffrey Managing the commitment to protect children from maltreatment : the case of child contact centres in England , 2014 .

[39]  Gerald Midgley,et al.  Systems thinking for evaluation , 2007 .

[40]  R. Kramer A Third Sector in the Third Millennium? , 2000 .

[41]  Lois Quam,et al.  The Audit Society: Rituals of Verification , 1998 .

[42]  M. Dixon-Woods,et al.  What is the role of individual accountability in patient safety? A multi‐site ethnographic study , 2015, Sociology of health & illness.

[44]  Jens Rasmussen,et al.  The role of error in organizing behaviour* , 1990 .

[45]  Vasilios K. Kimiskidis,et al.  Introduction , 2019, Int. J. Neural Syst..

[46]  C. McKevitt,et al.  Embedding research in health systems: lessons from complexity theory , 2016, Health Research Policy and Systems.

[47]  D. Woods Perspectives on Human Error: Hindsight Biases and Local Rationality , 2000 .

[48]  C. Harrison,et al.  SAFETY AND CHILD CONTACT: an analysis of the role of child contact centres in the context of domestic violence and child welfare concerns , 2002 .

[49]  V. Symonenko GOVERNANCE AND PUBLIC ADMINISTRATION , 2018, Economic innovations.

[50]  Eileen Munro,et al.  Improving practice: Child protection as a systems problem , 2005 .

[51]  C. Harrison Implacably Hostile or Appropriately Protective? , 2008, Violence against women.

[52]  E. Munro The Impact of Audit on Social Work Practice , 2004 .

[53]  Peter J Pronovost,et al.  Balancing "no blame" with accountability in patient safety. , 2009, The New England journal of medicine.

[54]  David H Peters,et al.  The application of systems thinking in health: why use systems thinking? , 2014, Health Research Policy and Systems.

[55]  Kirsten A. Gronbjerg,et al.  : Partners in Public Service: Government-Nonprofit Relations in the Modern Welfare State , 1997 .

[56]  J. Horwath Maintaining a focus on the child? First impressions of the Framework for the Assessment of Children in Need and their Families in cases of child neglect , 2002 .

[57]  Carol H. Weiss,et al.  Nothing as Practical as Good Theory : Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families , 2011 .

[58]  J Rasmussen,et al.  Human error and the problem of causality in analysis of accidents. , 1990, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[59]  Peter C. Smith,et al.  Outcome‐related Performance Indicators and Organizational Control in the Public Sector1 , 1993 .

[60]  Herbert A. Simon,et al.  The Sciences of the Artificial , 1970 .

[61]  Mae Keary,et al.  The Science of Evaluation: A Realist Manifesto , 2014, Online Inf. Rev..

[62]  Sarah Nichols,et al.  Cognitive Work Investigation and Design in Practice: The Influence of Social Context and Social Work Artefacts : Sarah Jackson, Sarah Nichols , 2003 .

[63]  P. Maurette [To err is human: building a safer health system]. , 2002, Annales francaises d'anesthesie et de reanimation.

[64]  I. Svedung,et al.  Proactive Risk Management in a Dynamic Society , 2000 .

[65]  Jens Rasmussen,et al.  Risk management in a dynamic society: a modelling problem , 1997 .

[66]  M. Okoye Social Care Institute for Excellence (SCIE) , 2010 .

[67]  J. Reason Human error: models and management , 2000, BMJ : British Medical Journal.

[68]  Erik Hollnagel,et al.  Joint Cognitive Systems: Patterns in Cognitive Systems Engineering , 2006 .

[69]  I. Sanderson,et al.  Evaluation in Complex Policy Systems , 2000 .

[70]  Eileen Munro,et al.  A Systems Approach to Investigating Child Abuse Deaths , 2005 .

[71]  J. Reason Human error: models and management , 2000, BMJ : British Medical Journal.

[72]  Fadhel Kaboub Realistic Evaluation , 2004 .

[73]  Gill Callaghan,et al.  Evaluation and Negotiated Order , 2008 .

[74]  C. May A rational model for assessing and evaluating complex interventions in health care , 2006, BMC Health Services Research.

[75]  Sarah Nichols,et al.  Cognitive Work Investigation and Design in Practice: The Influence of Social Context and Social Work Artefacts , 2003 .

[76]  Richard I. Cook,et al.  Nine Steps to Move Forward from Error , 2002, Cognition, Technology & Work.

[77]  Eileen Munro,et al.  What Tools Do We Need to Improve Identification of Child Abuse , 2005 .

[78]  S. Dekker,et al.  Balancing "no blame" with accountability in patient safety. , 2010, The New England journal of medicine.