TY - JOUR
T1 - Assess and validate predictive performance of models for in-hospital mortality in COVID-19 patients
T2 - A retrospective cohort study in the Netherlands comparing the value of registry data with high-granular electronic health records
AU - Vagliano, Iacopo
AU - Schut, Martijn C.
AU - Abu-Hanna, Ameen
AU - Dongelmans, Dave A.
AU - de Lange, Dylan W.
AU - Gommers, Diederik
AU - Cremer, Olaf L.
AU - Bosman, Rob J.
AU - Rigter, Sander
AU - Wils, Evert-Jan
AU - Frenzel, Tim
AU - de Jong, Remko
AU - Peters, Marco A. A.
AU - Kamps, Marlijn J. A.
AU - Ramnarain, Dharmanand
AU - Nowitzky, Ralph
AU - Nooteboom, Fleur G. C. A.
AU - de Ruijter, Wouter
AU - Urlings-Strop, Louise C.
AU - Smit, Ellen G. M.
AU - Mehagnoul-Schipper, D. Jannet
AU - Dormans, Tom
AU - de Jager, Cornelis P. C.
AU - Hendriks, Stefaan H. A.
AU - Achterberg, Sefanja
AU - Oostdijk, Evelien
AU - Reidinga, Auke C.
AU - Festen-Spanjer, Barbara
AU - Brunnekreef, Gert B.
AU - Cornet, Alexander D.
AU - van den Tempel, Walter
AU - Boelens, Age D.
AU - Koetsier, Peter
AU - Lens, Judith
AU - Faber, Harald J.
AU - Karakus, A.
AU - Entjes, Robert
AU - de Jong, Paul
AU - Rettig, Thijs C. D.
AU - Reuland, M. C.
AU - Arbous, Sesmu
AU - Fleuren, Lucas M.
AU - Dam, Tariq A.
AU - Thoral, Patrick J.
AU - Lalisang, Robbert C. A.
AU - Tonutti, Michele
AU - de Bruin, Daan P.
AU - Elbers, Paul W. G.
AU - de Keizer, Nicolette F.
N1 - Funding Information: We thank Sylvia Brinkman for her support with the extraction and understanding of the NICE data. The study protocol was reviewed by the Medical Ethics Committee of the Amsterdam Medical Center, the Netherlands. This committee provided a waiver from formal approval (W20_273 # 20.308) and informed consent since this trial does not fall within the scope of the Dutch Medical Research (Human Subjects) Act. This research was funded by The Netherlands Organisation for Health Research and Development (ZonMw) COVID-19 Programme in the bottom-up focus area 1 “Predictive diagnostics and treatment” for theme 3 “Risk analysis and prognostics” (project number 10430 01 201 0011: IRIS). The funder had no role in the design of the study or writing the manuscript. All participating hospitals have access to the Dutch ICU Data Warehouse and NICE data. The NICE registry data are available under conditions as described on the NICE website at stichting-nice.nl/extractieverzoek_procedure.jsp (in Dutch). External researchers can get access to the Dutch ICU Data Warehouse in collaboration with any of the participating hospitals. The list of collaborators is available in the co-author list and in the collaborators section, through the corresponding author, and through the contact details on amsterdammedicaldatascience.nl. Research questions have to be in line with the DSA; to investigate the course of COVID-19 in the ICU and to research potential treatments. Researchers have sign a code of conduct before accessing the data. The code used for our analyses is publicly available at bitbucket.org/aumc-kik/automl4covid. Funding Information: This research was funded by The Netherlands Organisation for Health Research and Development (ZonMw) COVID-19 Programme in the bottom-up focus area 1 “Predictive diagnostics and treatment” for theme 3 “Risk analysis and prognostics” (project number 10430 01 201 0011: IRIS). The funder had no role in the design of the study or writing the manuscript. Publisher Copyright: © 2022 The Author(s)
PY - 2022/11/1
Y1 - 2022/11/1
N2 - Purpose: To assess, validate and compare the predictive performance of models for in-hospital mortality of COVID-19 patients admitted to the intensive care unit (ICU) over two different waves of infections. Our models were built with high-granular Electronic Health Records (EHR) data versus less-granular registry data. Methods: Observational study of all COVID-19 patients admitted to 19 Dutch ICUs participating in both the national quality registry National Intensive Care Evaluation (NICE) and the EHR-based Dutch Data Warehouse (hereafter EHR). Multiple models were developed on data from the first 24 h of ICU admissions from February to June 2020 (first COVID-19 wave) and validated on prospective patients admitted to the same ICUs between July and December 2020 (second COVID-19 wave). We assessed model discrimination, calibration, and the degree of relatedness between development and validation population. Coefficients were used to identify relevant risk factors. Results: A total of 1533 patients from the EHR and 1563 from the registry were included. With high granular EHR data, the average AUROC was 0.69 (standard deviation of 0.05) for the internal validation, and the AUROC was 0.75 for the temporal validation. The registry model achieved an average AUROC of 0.76 (standard deviation of 0.05) in the internal validation and 0.77 in the temporal validation. In the EHR data, age, and respiratory-system related variables were the most important risk factors identified. In the NICE registry data, age and chronic respiratory insufficiency were the most important risk factors. Conclusion: In our study, prognostic models built on less-granular but readily-available registry data had similar performance to models built on high-granular EHR data and showed similar transportability to a prospective COVID-19 population. Future research is needed to verify whether this finding can be confirmed for upcoming waves.
AB - Purpose: To assess, validate and compare the predictive performance of models for in-hospital mortality of COVID-19 patients admitted to the intensive care unit (ICU) over two different waves of infections. Our models were built with high-granular Electronic Health Records (EHR) data versus less-granular registry data. Methods: Observational study of all COVID-19 patients admitted to 19 Dutch ICUs participating in both the national quality registry National Intensive Care Evaluation (NICE) and the EHR-based Dutch Data Warehouse (hereafter EHR). Multiple models were developed on data from the first 24 h of ICU admissions from February to June 2020 (first COVID-19 wave) and validated on prospective patients admitted to the same ICUs between July and December 2020 (second COVID-19 wave). We assessed model discrimination, calibration, and the degree of relatedness between development and validation population. Coefficients were used to identify relevant risk factors. Results: A total of 1533 patients from the EHR and 1563 from the registry were included. With high granular EHR data, the average AUROC was 0.69 (standard deviation of 0.05) for the internal validation, and the AUROC was 0.75 for the temporal validation. The registry model achieved an average AUROC of 0.76 (standard deviation of 0.05) in the internal validation and 0.77 in the temporal validation. In the EHR data, age, and respiratory-system related variables were the most important risk factors identified. In the NICE registry data, age and chronic respiratory insufficiency were the most important risk factors. Conclusion: In our study, prognostic models built on less-granular but readily-available registry data had similar performance to models built on high-granular EHR data and showed similar transportability to a prospective COVID-19 population. Future research is needed to verify whether this finding can be confirmed for upcoming waves.
KW - Covid-19 [C01.748.610.763.500]
KW - Critical care [E02.760.190]
KW - Electronic Health Record [E05.318.308.940.968.625.500]
KW - In-hospital mortality [E05.318.308.985.550.400]
KW - Machine learning [G17.035.250.500]
KW - Prognosis [E01.789]
UR - http://www.scopus.com/inward/record.url?scp=85138464263&partnerID=8YFLogxK
U2 - https://doi.org/10.1016/j.ijmedinf.2022.104863
DO - https://doi.org/10.1016/j.ijmedinf.2022.104863
M3 - Article
C2 - 36162166
SN - 1386-5056
VL - 167
SP - 104863
JO - International Journal of Medical Informatics
JF - International Journal of Medical Informatics
M1 - 104863
ER -