• Patrick Rockenschaub, Adam Hilbert, Tabea Kossen, Paul Elbers, Falk von Dincklage, Vince Istvan Madai, and Dietmar Frey.
• Charité Lab for Artificial Intelligence in Medicine (CLAIM), CharitéUniversitätsmedizin Berlin, Berlin, Germany .
• Crit. Care Med. 2024 Nov 1; 52 (11): 171017211710-1721.

ObjectivesTo evaluate the transferability of deep learning (DL) models for the early detection of adverse events to previously unseen hospitals.DesignRetrospective observational cohort study utilizing harmonized intensive care data from four public datasets.SettingICUs across Europe and the United States.PatientsAdult patients admitted to the ICU for at least 6 hours who had good data quality.InterventionsNone.Measurements And Main ResultsUsing carefully harmonized data from a total of 334,812 ICU stays, we systematically assessed the transferability of DL models for three common adverse events: death, acute kidney injury (AKI), and sepsis. We tested whether using more than one data source and/or algorithmically optimizing for generalizability during training improves model performance at new hospitals. We found that models achieved high area under the receiver operating characteristic (AUROC) for mortality (0.838-0.869), AKI (0.823-0.866), and sepsis (0.749-0.824) at the training hospital. As expected, AUROC dropped when models were applied at other hospitals, sometimes by as much as -0.200. Using more than one dataset for training mitigated the performance drop, with multicenter models performing roughly on par with the best single-center model. Dedicated methods promoting generalizability did not noticeably improve performance in our experiments.ConclusionsOur results emphasize the importance of diverse training data for DL-based risk prediction. They suggest that as data from more hospitals become available for training, models may become increasingly generalizable. Even so, good performance at a new hospital still depended on the inclusion of compatible hospitals during training.Copyright © 2024 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine and Wolters Kluwer Health, Inc.

23 keywords...

hide…