Derivation and Internal Validation of a Health Administrative Data Algorithm for Identifying Sepsis

Abstract

Objectives: The primary objective was to externally validate a Canadian ICD-coded sepsis case definition developed by Jolley et al. by examining its sensitivity and specificity in health administrative data among intensive care unit (ICU) patients at The Ottawa Hospital (TOH). We also evaluated whether two modifications (the addition of antimicrobial information and the removal of infection-related ICD codes) to the Jolley algorithm could improve sensitivity while maintaining specificity. The secondary objective was to examine the impact of these modifications on the algorithm's positive and negative predictive values.

Methods: We identified adults (18 years and older) admitted to TOH's Civic and General campus ICUs between April 1, 2014 and March 31, 2019. From this population, a random sample of 870 medical charts was selected for manual chart review, with sepsis positive and negative reference cases classified according to the Sepsis-3 criteria. Chart review data were linked to a health administrative dataset, containing pharmacy records, diagnosis and procedure codes. We then evaluated the sensitivity and specificity of the Jolley algorithm and its modified versions incorporating antimicrobial information and/or excluding infection codes.

Results: Among 10,407 eligible ICU patients, 833 of the 870 charts met the eligibility criteria. Of these, 391 patients met Sepsis-3 criteria through chart review, while 364 patients met Jolley sepsis criteria. The original Jolley algorithm had a sensitivity of 72.6% (95% CI: 69.6% - 75.7%) and specificity of 81.9% (95% CI: 79.3% - 84.5%). Incorporating antimicrobial information increased sensitivity to a range of 80.8% (95% CI: 78.1% - 83.5%) to 99.5% (95% CI: 99.0% -100.0%), but resulted in a marked decline in specificity, ranging from 12.2% (10.0% -14.4%) to 39.6% (95% CI: 36.3% - 42.9%). Removing infection-related ICD codes increased sensitivity to 83.4% (95% CI: 80.8% - 85.9%) but further reduced specificity to 26.9% (95% CI: 23.9% - 29.9%). Combining both modifications raised sensitivity to a range of 88.8% (95% CI: 86.6% - 90.9%) to 99.7% (95% CI: 99.4% -100.0%), while specificity dropped to 4.8% (95% CI: 3.3% - 6.2%) to 13.6% (95%CI: 11.2% -15.9%).

Conclusion: The Jolley algorithm demonstrated reasonable sensitivity and specificity in identifying sepsis patients among TOH ICU patients using administrative data. While adding antimicrobial information or removing infection codes substantially increased sensitivity, these modifications led to a marked decline in specificity. These findings highlight both the promise and limitations of using administrative data for sepsis surveillance and underscore the need for careful algorithm refinement to balance accuracy and utility.