TY - JOUR
T1 - Diagnostic stewardship for blood cultures in the emergency department
T2 - A multicenter validation and prospective evaluation of a machine learning prediction tool
AU - Schinkel, Michiel
AU - Boerman, Anneroos W.
AU - Bennis, Frank C.
AU - Minderhoud, Tanca C.
AU - Lie, Mei
AU - Peters-Sengers, Hessel
AU - Holleman, Frits
AU - Schade, Rogier P.
AU - de Jonge, Robert
AU - Wiersinga, W. Joost
AU - Nanayakkara, Prabath W. B.
N1 - Funding Information: We want to thank Melchior Pot (business intelligence specialist at Amsterdam UMC) for his help in implementing the prediction model in the EHR environment of the VUMC for the prospective evaluation. Furthermore, we thank Muhammad Al-Dulaimy, Bas Nieuwenhuizen, and Yvonne Bandt for providing the data from the ZMC. We are grateful for the financial contributions to this project made by the Amsterdam Public Health – Quality of Care program and the “Doen of Laten” project. Publisher Copyright: © 2022 The Author(s)
PY - 2022/8/1
Y1 - 2022/8/1
N2 - Background: Overuse of blood cultures (BCs) in emergency departments (EDs) leads to low yields and high numbers of contaminated cultures, accompanied by increased diagnostics, antibiotic usage, prolonged hospitalization, and mortality. We aimed to simplify and validate a recently developed machine learning model to help safely withhold BC testing in low-risk patients. Methods: We extracted data from the electronic health records (EHR) for 44.123 unique ED visits with BC sampling in the Amsterdam UMC (locations VUMC and AMC; the Netherlands), Zaans Medical Center (ZMC; the Netherlands), and Beth Israel Deaconess Medical Center (BIDMC; United States) in periods between 2011 and 2021. We trained a machine learning model on the VUMC data to predict blood culture outcomes and validated it in the AMC, ZMC, and BIDMC with subsequent real-time prospective evaluation in the VUMC. Findings: The model had an Area Under the Receiver Operating Characteristics curve (AUROC) of 0.81 (95%-CI = 0.78–0.83) in the VUMC test set. The most important predictors were temperature, creatinine, and C-reactive protein. The AUROCs in the validation cohorts were 0.80 (AMC; 0.78–0.82), 0.76 (ZMC; 0.74–0.78), and 0.75 (BIDMC; 0.74–0.76). During real-time prospective evaluation in the EHR of the VUMC, it reached an AUROC of 0.76 (0.71–0.81) among 590 patients with BC draws in the ED. The prospective evaluation showed that the model can be used to safely withhold blood culture analyses in at least 30% of patients in the ED. Interpretation: We developed a machine learning model to predict blood culture outcomes in the ED, which retained its performance during external validation and real-time prospective evaluation. Our model can identify patients at low risk of having a positive blood culture. Using the model in practice can significantly reduce the number of blood culture analyses and thus avoid the hidden costs of false-positive culture results. Funding: This research project was funded by the Amsterdam Public Health – Quality of Care program and the Dutch “Doen of Laten” project (project number: 839205002).
AB - Background: Overuse of blood cultures (BCs) in emergency departments (EDs) leads to low yields and high numbers of contaminated cultures, accompanied by increased diagnostics, antibiotic usage, prolonged hospitalization, and mortality. We aimed to simplify and validate a recently developed machine learning model to help safely withhold BC testing in low-risk patients. Methods: We extracted data from the electronic health records (EHR) for 44.123 unique ED visits with BC sampling in the Amsterdam UMC (locations VUMC and AMC; the Netherlands), Zaans Medical Center (ZMC; the Netherlands), and Beth Israel Deaconess Medical Center (BIDMC; United States) in periods between 2011 and 2021. We trained a machine learning model on the VUMC data to predict blood culture outcomes and validated it in the AMC, ZMC, and BIDMC with subsequent real-time prospective evaluation in the VUMC. Findings: The model had an Area Under the Receiver Operating Characteristics curve (AUROC) of 0.81 (95%-CI = 0.78–0.83) in the VUMC test set. The most important predictors were temperature, creatinine, and C-reactive protein. The AUROCs in the validation cohorts were 0.80 (AMC; 0.78–0.82), 0.76 (ZMC; 0.74–0.78), and 0.75 (BIDMC; 0.74–0.76). During real-time prospective evaluation in the EHR of the VUMC, it reached an AUROC of 0.76 (0.71–0.81) among 590 patients with BC draws in the ED. The prospective evaluation showed that the model can be used to safely withhold blood culture analyses in at least 30% of patients in the ED. Interpretation: We developed a machine learning model to predict blood culture outcomes in the ED, which retained its performance during external validation and real-time prospective evaluation. Our model can identify patients at low risk of having a positive blood culture. Using the model in practice can significantly reduce the number of blood culture analyses and thus avoid the hidden costs of false-positive culture results. Funding: This research project was funded by the Amsterdam Public Health – Quality of Care program and the Dutch “Doen of Laten” project (project number: 839205002).
KW - Blood cultures
KW - Decision-curve analysis
KW - Emergency department
KW - Machine learning
KW - Validation
UR - http://www.scopus.com/inward/record.url?scp=85134624873&partnerID=8YFLogxK
U2 - https://doi.org/10.1016/j.ebiom.2022.104176
DO - https://doi.org/10.1016/j.ebiom.2022.104176
M3 - Article
C2 - 35853298
SN - 2352-3964
VL - 82
JO - eBioMedicine
JF - eBioMedicine
M1 - 104176
ER -