The effect of improving task representativeness on capturing nurses’ risk assessment judgements: a comparison of written case simulations and physical simulations

BackgroundThe validity of studies describing clinicians’ judgements based on their responses to paper cases is questionable, because - commonly used - paper case simulations only partly reflect real clinical environments. In this study we test whether paper case simulations evoke similar risk assessment judgements to the more realistic simulated patients used in high fidelity physical simulations.Methods97 nurses (34 experienced nurses and 63 student nurses) made dichotomous assessments of risk of acute deterioration on the same 25 simulated scenarios in both paper case and physical simulation settings. Scenarios were generated from real patient cases. Measures of judgement ‘ecology’ were derived from the same case records. The relationship between nurses’ judgements, actual patient outcomes (i.e. ecological criteria), and patient characteristics were described using the methodology of judgement analysis. Logistic regression models were constructed to calculate Lens Model Equation parameters. Parameters were then compared between the modeled paper-case and physical-simulation judgements.ResultsParticipants had significantly less achievement (ra) judging physical simulations than when judging paper cases. They used less modelable knowledge (G) with physical simulations than with paper cases, while retaining similar cognitive control and consistency on repeated patients. Respiration rate, the most important cue for predicting patient risk in the ecological model, was weighted most heavily by participants.ConclusionsTo the extent that accuracy in judgement analysis studies is a function of task representativeness, improving task representativeness via high fidelity physical simulations resulted in lower judgement performance in risk assessments amongst nurses when compared to paper case simulations. Lens Model statistics could prove useful when comparing different options for the design of simulations used in clinical judgement analysis. The approach outlined may be of value to those designing and evaluating clinical simulations as part of education and training strategies aimed at improving clinical judgement and reasoning.

[1]  T. R. Brown.  A comparison of judgmental policy equations obtained from human judges under natural and contrived conditions , 1972 .

[2]  C. Franklin,et al.  Developing strategies to prevent inhospital cardiac arrest: Analyzing responses of physicians and nurses in the hours before the event , 1994, Critical care medicine.

[3]  Tracey Bucknall,et al.  Nurses' critical event risk assessments: a judgement analysis. , 2009, Journal of clinical nursing.

[4]  T. R. Stewart.  Chapter 2 Judgment Analysis: Procedures , 1988 .

[5]  P. Hoffman.  The paramorphic representation of clinical judgment. , 1960, Psychological bulletin.

[6]  S. I. Kramer,et al.  Using case vignettes to train clinicians and utilization reviewers to make level-of-care decisions. , 2000, Psychiatric services.

[7]  W. Bond,et al.  P REHOSPITAL AND H OSPITAL-BASED H EALTH C ARE P ROVIDERS ' E XPERIENCE WITH A H UMAN P ATIENT S IMULATOR , 2001, Prehospital emergency care : official journal of the National Association of EMS Physicians and the National Association of State EMS Directors.

[8]  J. Shanteau,et al.  Livestock judges: How much information can an expert use? , 1978 .

[9]  Mandeep K. Dhami,et al.  The role of representative design in an ecological approach to cognition. , 2004, Psychological bulletin.

[10]  S. Menard.  Applied Logistic Regression Analysis , 1996 .

[11]  H. Plessner,et al.  Intuition in judgment and decision making , 2008 .

[12]  R. Hamm.  Cue by hypothesis interactions in descriptive modeling of unconscious use of multiple intuitive judgment strategies , 2011 .

[13]  T. R. Stewart,et al.  Judgment Analysis: Procedures* , 2022 .

[14]  R. John.  Linear Statistical Models: An Applied Approach , 1986 .

[15]  B. Brehmer,et al.  Chapter 3 What Have we Learned about Human Judgment from Thirty Years of Policy Capturing , 1988 .

[16]  B. Brehmer,et al.  Human judgment : the SJT view , 1988 .

[17]  D M Chaput de Saintonge,et al.  Clinical judgment in rheumatoid arthritis. I. Rheumatologists' opinions and the development of 'paper patients'. , 1983, Annals of the rheumatic diseases.

[18]  Robert Tibshirani,et al.  Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy , 1986 .

[19]  K R Hammond,et al.  Computer Graphics as an Aid to Learning , 1971, Science.

[20]  Jacob Cohen.  A Coefficient of Agreement for Nominal Scales , 1960 .

[21]  E. Brunswik,et al.  The Conceptual Framework of Psychology , 1954 .

[22]  L. Delbeke.  Quasi-experimentation - design and analysis issues for field settings - cook,td, campbell,dt , 1980 .

[23]  T. Pearson,et al.  Direct comparison of the efficacy of intuitive and analytical cognition in expert judgment , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[24]  Michael E. Doherty,et al.  Social judgement theory , 1996 .

[25]  D. Tritchler,et al.  Use of written cases to study factors associated with regional variations in referral rates. , 1991, Journal of clinical epidemiology.

[26]  E. Brunswik,et al.  The psychology of Egon Brunswik , 1966 .

[27]  Hans C. Jessen,et al.  Applied Logistic Regression Analysis , 1996 .

[28]  Michael Grüninger,et al.  Introduction , 2002, CACM.

[29]  Robert S. Wigton,et al.  Social Judgement Theory and Medical Judgement , 1996 .

[30]  M. Greenwood.  An Introduction to Medical Statistics , 1932, Nature.

[31]  T. Hodgetts,et al.  Incidence, location and reasons for avoidable in-hospital cardiac arrest in a district general hospital. , 2002, Resuscitation.

[32]  E. Ziegel,et al.  Bootstrapping: A Nonparametric Approach to Statistical Inference , 1993 .

[33]  M S Gerrity,et al.  Written case simulations: do they predict physicians' behavior? , 1990, Journal of clinical epidemiology.

[34]  R S Wigton,et al.  How Physicians Use Clinical Information in Diagnosing Pulmonary Embolism , 1986, Medical decision making : an international journal of the Society for Medical Decision Making.

[35]  M K Campbell,et al.  Use of unequal randomisation to aid the economic efficiency of clinical trials , 2000, BMJ : British Medical Journal.

[36]  J W Peabody,et al.  Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality. , 2000, JAMA.

[37]  Kenneth R Hammond,et al.  Judgement and Decision Making in Dynamic Tasks , 1988 .

[38]  D. M. Chaput de Saintonge,et al.  Antibiotic use in otitis media: patient simulations as an aid to audit. , 1981, British medical journal.

[39]  M. Cohen,et al.  The Validity of Performance Assessments Using Simulation , 2001, Anesthesiology.

[40]  J. Scott Armstrong,et al.  Principles of forecasting : a handbook for researchers and practitioners , 2001 .

[41]  K Rich,et al.  Inhospital cardiac arrest: pre-event variables and nursing response. , 1999, Clinical nurse specialist CNS.

[42]  Andrea Wyatt,et al.  Use of simulators in teaching and learning: Paramedics' evaluation of a Patient Simulator? , 2015 .

[43]  T. R. Stewart,et al.  Judgment and Decision Making in Dynamic Tasks: The Case of Forecasting the Microburst , 1990 .

[44]  Thomas R. Stewart,et al.  Improving Reliability of Judgmental Forecasts , 2001 .

[45]  F McArthur-Rouse,et al.  Critical care outreach services and early warning scoring systems: a review of the literature. , 2001, Journal of advanced nursing.

[46]  Ray W. Cooksey,et al.  Judgment analysis : theory, methods, and applications , 1996 .

[47]  Dorothy McCaughan,et al.  Nurses, information use, and clinical decision making—the real world potential for evidence-based decisions in nursing , 2004, Evidence-based nursing.

[48]  T. Cook,et al.  Quasi-experimentation: Design & analysis issues for field settings , 1979 .

[49]  C. Subbe,et al.  Validation of a modified Early Warning Score in medical admissions. , 2001, QJM : monthly journal of the Association of Physicians.

[50]  K Doggen,et al.  Information sources used in decision making: considerations for simulation development. , 1996, International journal of nursing studies.

[51]  M. Roland,et al.  Analysis of referral behaviour: responses to simulated case histories may not reflect real clinical behaviour. , 1990, The British journal of general practice : the journal of the Royal College of General Practitioners.

[52]  D. Rovner,et al.  Methods of Analyzing Physician Practice Patterns in Hypertension , 1989, Medical care.

[53]  Andrew S. Glassner.  Upon Reflection , 1998, IEEE Computer Graphics and Applications.

[54]  T. R. Stewart,et al.  The essential brunswik: Beginnings, explications, applications. , 2001 .

[55]  V. Farewell,et al.  Conditional inference for subject-specific and marginal agreement: Two families of agreement measures† , 1995 .