The Epic Sepsis Model (ESM) used in several hospital systems across the United States was found to be a poor predictor of sepsis due to its low sensitivity, inadequate calibration and discrimination, and tendency to cause alert fatigue, according to authors of a study published in JAMA Internal Medicine.
Using electronic health record data from the Michigan Medicine health system, this retrospective, external validation cohort study evaluated the effectiveness of the ESM in alerting clinicians to the onset of sepsis. Patients enrolled (N=27,697) were 18 years of age and older, with 38,455 recorded hospitalizations between December 6, 2018, and October 20, 2019.
ESM scores, prospectively calculated by the model every 15 minutes after the patient arrived at the hospital, were used to predict sepsis onset. ESM scores calculated after a patient experienced sepsis were excluded. An ESM score of 6 was set as the sepsis risk threshold for this study and in accordance with the standard practices of Michigan Medicine, the academic health system of the University of Michigan. Additionally, to assess alert fatigue and the clinical benefit of the ESM, the investigators evaluated the time between a patient reaching a score of 6 and the initiation of antibiotic therapy.
Continue Reading
Results showed that of the 2552 patients identified as having sepsis based on an ESM score of 6 or greater, 183 (7%) were not started on antibiotic therapy within 3 hours. In 18% (n=6971) of all hospitalizations in which patients scored 6 or greater, ESM triggered alerts to possible sepsis.
The hospitalization-level performance of ESM (area under curve [AUC], 0.63; 95% CI, 0.62-0.64) was substantially worse than that reported by its developer, Epic Systems (AUC, 0.76-0.83). Time horizon-based AUCs were higher (AUC, 0.72-0.76), although the study authors noted they “are misleading because they treat each prediction as independent.”
In a sensitivity analysis, the hospitalization-level AUC increased when the investigators included scores taken from the point at which sepsis occurred up to 3 hours after its onset (AUC, 0.80; 95% CI, 0.79-0.81). Furthermore, at the ESM score threshold of 6, specificity was 83%, hospitalization-level sensitivity was 33%, positive predictive value was 12%, and negative predictive value was 95%.
Median lead time between an ESM score first exceeding 6 was 2.5 hours (interquartile range, 0.5-15.6 h). Overall, inadequate calibration of the ESM was noted; 67% of patients with sepsis (n=1709) were not recognized by the ESM, despite alerts being generated for 18% of all hospitalized patients, “thus creating a large burden of alert fatigue,” according to the investigators. In 60% of these patients, however, antibiotic therapy was initiated on time.
According to the authors, large variations in reported AUC may be due to consideration of sepsis timing, and previous research has suggested that ESM-driven alerts reflect sepsis when it is already apparent to clinicians. The investigators believe their study “has important national implications” and urge that “[m]edical professional organizations constructing national guidelines should be cognizant of the broad use of these algorithms and make formal recommendations about their use.”
Reference
Wong A, Otles E, Donnelly JP, et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern Med. Published online June 21, 2021. doi:10.1001/jamainternmed.2021.2626
This article originally appeared on Infectious Disease Advisor
