Validation of Automated Data Extraction From the Electronic Medical Record to Provide a Pediatric Risk Assessment Score

BACKGROUND: Although the rate of pediatric postoperative mortality is low, the development and validation of perioperative risk assessment models have allowed for the stratification of those at highest risk, including the Pediatric Risk Assessment (PRAm) score. The clinical application of such tools requires manual data entry, which may be inaccurate or incomplete, compromise efficiency, and increase physicians’ clerical obligations. We aimed to create an electronically derived, automated PRAm score and to evaluate its agreement with the original American College of Surgery National Surgical Quality Improvement Program (ACS NSQIP)-derived and validated score. METHODS: We performed a retrospective observational study of children <18 years who underwent noncardiac surgery from 2017 through 2021 at Boston Children’s Hospital (BCH). An automated PRAm score was developed via electronic derivation of International Classification of Disease (ICD)-9 and -10 codes. The primary outcome was agreement and correlation among PRAm scores obtained via automation, NSQIP data, and manual physician entry from the same BCH cohort. The secondary outcome was discriminatory ability of the 3 PRAm versions. Fleiss Kappa, Spearman correlation (rho), and intraclass correlation coefficient (ICC) and receiver operating characteristic (ROC) curve analyses with area under the curve (AUC) were applied accordingly. RESULTS: Of the 6014 patients with NSQIP and automated PRAm scores (manual scores: n = 5267), the rate of 30-day mortality was 0.18% (n = 11). Agreement and correlation were greater between the NSQIP and automated scores (rho = 0.78; 95% confidence interval [CI], 0.76–0.79; P <.001; ICC = 0.80; 95% CI, 0.79–0.81; Fleiss kappa = 0.66; 95% CI, 0.65–0.67) versus the NSQIP and manual scores (rho = 0.73; 95% CI, 0.71–0.74; P < .001; ICC = 0.78; 95% CI, 0.77–0.79; Fleiss kappa = 0.56; 95% CI, 0.54–0.57). ROC analysis with AUC showed the manual score to have the greatest discrimination (AUC = 0.976; 95% CI, 0.959,0.993) compared to the NSQIP (AUC = 0.904; 95% CI, 0.792–0.999) and automated (AUC = 0.880; 95% CI, 0.769–0.999) scores. CONCLUSIONS: Development of an electronically derived, automated PRAm score that maintains good discrimination for 30-day mortality in neonates, infants, and children after noncardiac surgery is feasible. The automated PRAm score may reduce the preoperative clerical workload and provide an efficient and accurate means by which to risk stratify neonatal and pediatric surgical patients with the goal of improving clinical outcomes and resource utilization.