• Critical care medicine · Nov 2024

    Multicenter Study Observational Study

    The Impact of Multi-Institution Datasets on the Generalizability of Machine Learning Prediction Models in the ICU.

    • Patrick Rockenschaub, Adam Hilbert, Tabea Kossen, Paul Elbers, Falk von Dincklage, Vince Istvan Madai, and Dietmar Frey.
    • Charité Lab for Artificial Intelligence in Medicine (CLAIM), CharitéUniversitätsmedizin Berlin, Berlin, Germany .
    • Crit. Care Med. 2024 Nov 1; 52 (11): 171017211710-1721.

    ObjectivesTo evaluate the transferability of deep learning (DL) models for the early detection of adverse events to previously unseen hospitals.DesignRetrospective observational cohort study utilizing harmonized intensive care data from four public datasets.SettingICUs across Europe and the United States.PatientsAdult patients admitted to the ICU for at least 6 hours who had good data quality.InterventionsNone.Measurements And Main ResultsUsing carefully harmonized data from a total of 334,812 ICU stays, we systematically assessed the transferability of DL models for three common adverse events: death, acute kidney injury (AKI), and sepsis. We tested whether using more than one data source and/or algorithmically optimizing for generalizability during training improves model performance at new hospitals. We found that models achieved high area under the receiver operating characteristic (AUROC) for mortality (0.838-0.869), AKI (0.823-0.866), and sepsis (0.749-0.824) at the training hospital. As expected, AUROC dropped when models were applied at other hospitals, sometimes by as much as -0.200. Using more than one dataset for training mitigated the performance drop, with multicenter models performing roughly on par with the best single-center model. Dedicated methods promoting generalizability did not noticeably improve performance in our experiments.ConclusionsOur results emphasize the importance of diverse training data for DL-based risk prediction. They suggest that as data from more hospitals become available for training, models may become increasingly generalizable. Even so, good performance at a new hospital still depended on the inclusion of compatible hospitals during training.Copyright © 2024 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine and Wolters Kluwer Health, Inc.

      Pubmed     Copy Citation     Plaintext  

      Add institutional full text...

    Notes

     
    Knowledge, pearl, summary or comment to share?
    300 characters remaining
    help        
    You can also include formatting, links, images and footnotes in your notes
    • Simple formatting can be added to notes, such as *italics*, _underline_ or **bold**.
    • Superscript can be denoted by <sup>text</sup> and subscript <sub>text</sub>.
    • Numbered or bulleted lists can be created using either numbered lines 1. 2. 3., hyphens - or asterisks *.
    • Links can be included with: [my link to pubmed](http://pubmed.com)
    • Images can be included with: ![alt text](https://bestmedicaljournal.com/study_graph.jpg "Image Title Text")
    • For footnotes use [^1](This is a footnote.) inline.
    • Or use an inline reference [^1] to refer to a longer footnote elseweher in the document [^1]: This is a long footnote..

    hide…