The development of an evaluation framework for injury surveillance systems

BackgroundAccess to good quality information from injury surveillance is essential to develop and monitor injury prevention activities. To determine if information obtained from surveillance is of high quality, the limitations and strengths of a surveillance system are often examined. Guidelines have been developed to assist in evaluating certain types of surveillance systems. However, to date, no standard guidelines have been developed to specifically evaluate an injury surveillance system. The aim of this research is to develop a framework to guide the evaluation of injury surveillance systems.MethodsThe development of an Evaluation Framework for Injury Surveillance Systems (EFISS) involved a four stage process. First, a literature review was conducted to identify an initial set of characteristics that were recognised as important and/or had been recommended to be assessed in an evaluation of a surveillance system. Second, this set of characteristics was assessed using SMART criteria. Third, those surviving were presented to an expert panel using a two round modified-Delphi study to gain an alternative perspective on characteristic definitions, practicality of assessment, and characteristic importance. Finally, a rating system was created for the EFISS characteristics.ResultsThe resulting EFISS consisted of 18 characteristics that assess three areas of an injury surveillance system – five characteristics assess data quality, nine characteristics assess the system's operation, and four characteristics assess the practical capability of an injury surveillance system. A rating system assesses the performance of each characteristic.ConclusionThe development of the EFISS builds upon existing evaluation guidelines for surveillance systems and provides a framework tailored to evaluate an injury surveillance system. Ultimately, information obtained through an evaluation of an injury data collection using the EFISS would be useful for agencies to recommend how a collection could be improved to increase its usefulness for injury surveillance and in the long-term injury prevention.

[1]  Arash Shahin,et al.  Prioritization of key performance indicators: An integration of analytical hierarchy process and goal setting , 2007 .

[2]  Stuart Vaughan Newstead,et al.  Comparative analysis of several vehicle safety rating systems , 2001 .

[3]  I. B. Pless,et al.  Evaluation of the quality of an injury surveillance system. , 1999, American journal of epidemiology.

[4]  K. Brockhoff,et al.  The Performance of Forecasting Groups in Computer Dialogue and Face-to-face Discussion , 2002 .

[5]  Lisa M. Lee,et al.  Principles & Practice of Public Health Surveillance , 2010 .

[6]  Rodney L. Custer,et al.  The Modified Delphi Technique - A Rotational Modification , 1999 .

[7]  S B Thacker,et al.  The Science of Public Health Surveillance , 1989, Journal of public health policy.

[8]  D. Sackett,et al.  Evidence based medicine: what it is and what it isn't , 1996, BMJ.

[9]  Y. Holder,et al.  Injury surveillance guidelines , 2001 .

[10]  A. Carter,et al.  Public health surveillance: historical origins, methods and evaluation. , 1994, Bulletin of the World Health Organization.

[11]  Thacker Sb,et al.  A method for evaluating systems of epidemiological surveillance. , 1988 .

[12]  C. Webb,et al.  The Delphi technique: a methodological discussion. , 1994, Journal of advanced nursing.

[13]  S B Thacker,et al.  Public health surveillance in the United States. , 1988, Epidemiologic reviews.

[14]  R. G. Parrish,et al.  Guidelines for evaluating surveillance systems. , 1988, MMWR supplements.

[15]  F. Rivara,et al.  Centers for Disease Control and Prevention injury research agenda: identification of acute care research topics of interest to the Centers for disease Control and Prevention--National Center for Injury Prevention and Control. , 2004, Journal of Trauma.

[16]  Cheryl Bagley Thompson,et al.  Comparison of mailed vs. Internet applications of the Delphi technique in clinical informatics research , 2000, AMIA.

[17]  W. Eylenbosch,et al.  Surveillance in health and disease , 1988 .

[18]  A. Detsky,et al.  Evidence-based medicine. A new approach to teaching the practice of medicine. , 1992, JAMA.

[19]  Ruth Gilbert,et al.  Assessing Diagnostic and Screening Tests , 2007 .

[20]  R. Brook,et al.  Consensus methods: characteristics and guidelines for use. , 1984, American journal of public health.

[21]  William Halperin,et al.  Public Health Surveillance , 2008 .

[22]  Martin Weber,et al.  Generally accepted rating principles: A primer , 2001 .

[23]  Overview of the WHO framework for monitoring and evaluating surveillance and response systems for communicable diseases. , 2004, Releve epidemiologique hebdomadaire.

[24]  H. Hedegaard,et al.  Firearm-related injury surveillance in Colorado. , 1998, American journal of preventive medicine.

[25]  Vincent‐Wayne Mitchell,et al.  The delphi technique: an exposition and application , 1991 .

[26]  Don H. Bristow,et al.  Tech Prep Programs: The Role And Essential Elements , 1997 .

[27]  M. Frommer,et al.  Criteria for evaluating evidence on public health interventions , 2002, Journal of epidemiology and community health.

[28]  Christopher J. L. Cunningham,et al.  Coaching leaders in middle and executive management: goals, performance, buy‐in , 2007 .

[29]  S B Sharkey,et al.  An approach to consensus building using the Delphi technique: developing a learning resource in mental health. , 2001, Nurse education today.

[30]  George Wright,et al.  The Delphi technique as a forecasting tool: issues and analysis , 1999 .

[31]  Frederick P Rivara,et al.  Management of the injured patient: identification of research topics for systematic review using the delphi technique. , 2003, The Journal of trauma.

[32]  F. Hasson,et al.  Research guidelines for the Delphi survey technique. , 2000, Journal of advanced nursing.

[33]  S. Groseclose,et al.  Evaluating Public Health Surveillance , 2010 .

[34]  A. Morrison,et al.  Developing injury surveillance in accident and emergency departments , 1998, Archives of disease in childhood.

[35]  Henry L. Tosi.  A Theory of Goal Setting and Task Performance , 1991 .

[36]  F. Rivara,et al.  Rehabilitation of Traumatic Injuries: Use of the Delphi Method to Identify Topics for Evidence-Based Review , 2003, American journal of physical medicine & rehabilitation.

[37]  Mark S. Carey,et al.  Credit risk rating systems at large US banks , 2000 .

[38]  W. E. Riggs,et al.  The Delphi technique: An experimental evaluation , 1983 .

[39]  S. Thacker,et al.  Planning a public health surveillance system. , 1995, Epidemiological bulletin.

[40]  V. Moyer.  Evidence based pediatrics and child health , 2000 .

[41]  T. Sumsion,et al.  The Delphi Technique: An Adaptive Research Tool , 1998 .

[42]  Steven M. Teutsch,et al.  Considerations in Planning a Surveillance System , 2010 .

[43]  G. Gordon,et al.  Surveillance of interpersonal violence in Kingston, Jamaica: an evaluation , 2002, Injury control and safety promotion.

[44]  Fatal firearm-related injury surveillance in Maryland. , 1998, American journal of preventive medicine.

[45]  Sue Mallonee,et al.  Injury surveillance. , 2003, Epidemiologic reviews.

[46]  S B Thacker,et al.  Future directions for comprehensive public health surveillance and health information systems in the United States. , 1994, American journal of epidemiology.

[47]  Simone J. van Zolingen,et al.  Selection processes in a Delphi study about key qualifications in Senior Secondary Vocational Education , 2003 .

[48]  Raymond T. Stefani,et al.  Survey of the major world sports rating systems , 1997 .

[49]  Philip L. Graitcer,et al.  The development of state and local injury surveillance systems , 1987 .

[50]  S B Thacker,et al.  A method for evaluating systems of epidemiological surveillance. , 1988, World health statistics quarterly. Rapport trimestriel de statistiques sanitaires mondiales.

[51]  L Svanström,et al.  Potential of using existing injury information for injury surveillance at the local level in developing countries: experiences from Bangladesh. , 2000, Public health.

[52]  R. McClure,et al.  The Australian Capital Territory Injury Surveillance and Prevention Project. , 1995, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[53]  L. Lee,et al.  Updated guidelines for evaluating public health surveillance systems: recommendations from the Guidelines Working Group. , 2001, MMWR. Recommendations and reports : Morbidity and mortality weekly report. Recommendations and reports.

[54]  G Rechnitzer,et al.  Development of the Safe Tractor Assessment Rating System. , 2005, Journal of agricultural safety and health.