Evaluating the Impact of Using GRASP Framework on Clinicians and Healthcare Professionals Decisions in Selecting Clinical Predictive Tools

Background. When selecting predictive tools, clinicians and healthcare professionals are challenged with an overwhelming number of tools, most of which have never been evaluated for comparative effectiveness. To overcome this challenge, the authors developed and validated an evidence-based framework for grading and assessment of predictive tools (GRASP), based on the critical appraisal of published evidence. Methods. To examine GRASP impact on professionals decisions, a controlled experiment was conducted through an online survey. Randomising two groups of tools and two scenarios; participants were asked to select the best tools; most validated or implemented, with and without GRASP. A wide group of international participants were invited. Task completion time, rate of correct decisions, rate of objective vs subjective decisions, and level of decisional conflict were measured. Results. Valid responses received were 194. Compared to not using the framework, GRASP significantly increased correct decisions by 64% (T=8.53, p<0.001), increased objective decision making by 32% (T=9.24, p<0.001), and decreased subjective decision making; based on guessing and based on prior knowledge or experience by 20% (T=-5.47, p<0.001) and 8% (T=-2.99, p=0.003) respectively. GRASP significantly decreased decisional conflict; increasing confidence and satisfaction of participants with their decisions by 11% (T=4.27, p<0.001) and 13% (T=4.89, p<0.001) respectively. GRASP decreased task completion time by 52% (T=-0.87, p=0.384). The average system usability scale of GRASP was very good; 72.5%, and 88% of participants found GRASP useful. Discussion and Conclusions. Using GRASP has positively supported and significantly improved evidence-based decision making and increased accuracy and efficiency of selecting predictive tools.

[1]  R. Derlet,et al.  Utilizing clinical factors to reduce head CT scan ordering for minor head trauma patients. , 1997, The Journal of emergency medicine.

[2]  N. Maskrey,et al.  How clinical decisions are made , 2012, British journal of clinical pharmacology.

[3]  Monique W. M. Jaspers,et al.  Clinicians satisfaction with CPOE ease of use and effect on clinicians' workflow, efficiency and medication safety , 2011, Int. J. Medical Informatics.

[4]  Gordon H Guyatt,et al.  GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction , 2016, British Medical Journal.

[5]  Ferdinand T. Velasco,et al.  Improving Outcomes with Clinical Decision Support: An Implementer's Guide , 2012 .

[6]  J. Chamberlain,et al.  Quality Improvement in Pediatric Head Trauma with PECARN Rules Implementation as Computerized Decision Support , 2017, Pediatric quality & safety.

[7]  G. Guyatt,et al.  GRADE: an emerging consensus on rating quality of evidence and strength of recommendations , 2008, BMJ : British Medical Journal.

[8]  Kathy Boutis,et al.  Should a Head-Injured Child Receive a Head CT Scan? A Systematic Review of Clinical Prediction Rules , 2009, Pediatrics.

[9]  N. Roberts,et al.  Evidence of Effectiveness of Health Care Professionals Using Handheld Computers: A Scoping Review of Systematic Reviews , 2013, Journal of medical Internet research.

[10]  I. Stiell,et al.  A clinical decision rule for cranial computed tomography in minor pediatric head trauma. , 2008, Archives of pediatrics & adolescent medicine.

[11]  George A Wells,et al.  Performance of the Canadian CT Head Rule and the New Orleans Criteria for predicting any traumatic intracranial injury on computed tomography in a United States Level I trauma center. , 2012, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[12]  B. Mol,et al.  Decision aids to improve informed decision‐making in pregnancy care: a systematic review , 2013, BJOG : an international journal of obstetrics and gynaecology.

[13]  E. Balas,et al.  Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success , 2005, BMJ : British Medical Journal.

[14]  R. Derlet,et al.  A decision rule for identifying children at low risk for brain injuries after blunt head trauma. , 2003, Annals of emergency medicine.

[15]  George A Wells,et al.  The Canadian CT Head Rule for patients with minor head injury , 2001, The Lancet.

[16]  Abdelkamel Tari,et al.  Exploring user satisfaction for e-learning systems via usage-based metrics and system usability scale analysis , 2016, Comput. Hum. Behav..

[17]  Christos Katsanos,et al.  Perceived usability evaluation of learning management systems: Empirical evaluation of the System Usability Scale , 2015 .

[18]  Marion Smits,et al.  External validation of the Canadian CT Head Rule and the New Orleans Criteria for CT scanning in patients with minor head injury. , 2005, JAMA.

[19]  T. Mills,et al.  Indications for computed tomography in patients with minor head injury. , 2000, The New England journal of medicine.

[20]  Paul A. Fontelo,et al.  Utilization of the PICO framework to improve searching PubMed for clinical questions , 2007, BMC Medical Informatics Decis. Mak..

[21]  G. Bedogni,et al.  Clinical Prediction Models—a Practical Approach to Development, Validation and Updating , 2009 .

[22]  P. Beattie,et al.  Clinical prediction rules: what are they and what do they tell us? , 2006, The Australian journal of physiotherapy.

[23]  Deirdre McCaughey,et al.  Rationality versus reality: the challenges of evidence-based decision making for health policy makers , 2010, Implementation science : IS.

[24]  J. B. Brooke,et al.  SUS: a retrospective , 2013 .

[25]  Janet Houser,et al.  Evidence-Based Practice: An Implementation Guide for Healthcare Organizations , 2010 .

[26]  J. Tu,et al.  Cardiovascular Disease Population Risk Tool (CVDPoRT): predictive algorithm for assessing CVD risk in the community setting. A study protocol , 2014, BMJ Open.

[27]  H. Mcdonald,et al.  Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. , 2005, JAMA.

[28]  Philip T. Kortum,et al.  Determining what individual SUS scores mean: adding an adjective rating scale , 2009 .

[29]  A. Parsaik,et al.  Evidence Based Public Health , 2013 .

[30]  Erik Frekjmr,et al.  Measuring Usability: Are Effectiveness, Efficiency, and Satisfaction Really Correlated? , 2000 .

[31]  Gari D. Clifford,et al.  Shortliffe Edward H, Cimino James J: "Biomedical Informatics; Computer Applications in Health Care and Biomedicine" , 2006 .

[32]  Edgar Erdfelder,et al.  G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences , 2007, Behavior research methods.

[33]  J. Harvey,et al.  The Kimberley Hospital Rule (KHR) for urgent computed tomography of the brain in a resource-limited environment. , 2013, South African medical journal = Suid-Afrikaanse tydskrif vir geneeskunde.

[34]  M. Ebell.  Evidence-Based Diagnosis: A Handbook of Clinical Prediction Rules , 2001 .

[35]  J. Dunning,et al.  Derivation of the children’s head injury algorithm for the prediction of important clinical events decision rule for head injury in children , 2006, Archives of Disease in Childhood.

[36]  Patrick Blake,et al.  Clinical decision support systems for improving diagnostic accuracy and achieving precision medicine , 2015, Journal of Clinical Bioinformatics.

[37]  Ephraim R. McLean,et al.  The DeLone and McLean Model of Information Systems Success: A Ten-Year Update , 2003, J. Manag. Inf. Syst..

[38]  H. Sox,et al.  Clinical prediction rules. Applications and methodological standards. , 1985, The New England journal of medicine.

[39]  B. Head.  Toward More “Evidence-Informed” Policy Making? , 2016 .

[40]  Sami Souissi,et al.  Prediction value of the Canadian CT head rule and the New Orleans criteria for positive head CT scan and acute neurosurgical procedures in minor head trauma: a multicenter external validation study. , 2013, Annals of emergency medicine.

[41]  S. Zanconato,et al.  Implementation of adapted PECARN decision rule for children with minor head injury in the pediatric emergency department. , 2012, Academic emergency medicine : official journal of the Society for Academic Emergency Medicine.

[42]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[43]  Mohamed Alfateh Badawy,et al.  Identification of children at very low risk of clinically-important brain injuries after head trauma: a prospective cohort study , 2009, The Lancet.

[44]  T J Ulahannan,et al.  Decision Making in Health and Medicine: Integrating Evidence and Values , 2002 .

[45]  Enrique Rubio,et al.  Reliability of clinical guidelines in the detection of patients at risk following mild head injury: results of a prospective study. , 2004, Journal of neurosurgery.

[46]  Andrew Muddimer,et al.  The effect of experience on system usability scale ratings , 2012 .

[47]  B. Zink,et al.  Pathophysiology of traumatic brain injury. , 2009, The Mount Sinai journal of medicine, New York.

[48]  S. Adams,et al.  Clinical prediction rules , 2012, BMJ : British Medical Journal.

[49]  Alberto De la Rosa Algaŕın.  Clinical Decision Support Systems in Biomedical Informatics and their Limitations , 2011 .

[50]  Rosalind Raine,et al.  Developing clinical guidelines: a challenge to current methods , 2005, BMJ : British Medical Journal.

[51]  P G Shekelle,et al.  Developing clinical guidelines. , 1999, The Western journal of medicine.

[52]  Farah Magrabi,et al.  Developing an Evidence-Based Framework for Grading and Assessment of Predictive Tools for Clinical Decision Support , 2019, ArXiv.

[53]  Johanna I. Westbrook,et al.  Do online information retrieval systems help experienced clinicians answer clinical questions? , 2005, Journal of the American Medical Informatics Association : JAMIA.

[54]  A. Rashidian,et al.  Guidelines for Guidelines: Are They Up to the Task? A Comparative Assessment of Clinical Practice Guideline Development Handbooks , 2012, PloS one.

[55]  Nathan Kuppermann,et al.  Clinical Prediction Rules for Children: A Systematic Review , 2011, Pediatrics.

[56]  Erwin Haasnoot,et al.  QRTEngine: An easy solution for running online reaction time experiments using Qualtrics , 2014, Behavior research methods.

[57]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[58]  M A Kish,et al.  Guide to development of practice guidelines. , 2001, Clinical infectious diseases : an official publication of the Infectious Diseases Society of America.

[59]  P. Shekelle,et al.  Systematic Review: Impact of Health Information Technology on Quality, Efficiency, and Costs of Medical Care , 2006, Annals of Internal Medicine.

[60]  C. L. Ventola.  Mobile devices and apps for health care professionals: uses and benefits. , 2014, P & T : a peer-reviewed journal for formulary management.

[61]  Rhonda Correll,et al.  CATCH: a clinical decision rule for the use of computed tomography in children with minor head injury , 2010, Canadian Medical Association Journal.