• Anne A H de Hond, Ilse M J Kant, Mattia Fornasa, Giovanni Cinà, ElbersPaul W GPWGDepartment of Intensive Care Medicine, Laboratory for Critical Care Computational Intelligence, Amsterdam UMC, Amsterdam, The Netherlands., Patrick J Thoral, Sesmu ArbousMMDepartment of Intensive Care Medicine, Leiden University Medical Centre, Leiden, The Netherlands., and Ewout W Steyerberg.
• Department of Information Technology and Digital Innovation, Leiden University Medical Centre, Leiden, The Netherlands.
• Crit. Care Med. 2023 Feb 1; 51 (2): 291300291-300.

ObjectivesMany machine learning (ML) models have been developed for application in the ICU, but few models have been subjected to external validation. The performance of these models in new settings therefore remains unknown. The objective of this study was to assess the performance of an existing decision support tool based on a ML model predicting readmission or death within 7 days after ICU discharge before, during, and after retraining and recalibration.DesignA gradient boosted ML model was developed and validated on electronic health record data from 2004 to 2021. We performed an independent validation of this model on electronic health record data from 2011 to 2019 from a different tertiary care center.SettingTwo ICUs in tertiary care centers in The Netherlands.PatientsAdult patients who were admitted to the ICU and stayed for longer than 12 hours.InterventionsNone.Measurements And Main ResultsWe assessed discrimination by area under the receiver operating characteristic curve (AUC) and calibration (slope and intercept). We retrained and recalibrated the original model and assessed performance via a temporal validation design. The final retrained model was cross-validated on all data from the new site. Readmission or death within 7 days after ICU discharge occurred in 577 of 10,052 ICU admissions (5.7%) at the new site. External validation revealed moderate discrimination with an AUC of 0.72 (95% CI 0.67-0.76). Retrained models showed improved discrimination with AUC 0.79 (95% CI 0.75-0.82) for the final validation model. Calibration was poor initially and good after recalibration via isotonic regression.ConclusionsIn this era of expanding availability of ML models, external validation and retraining are key steps to consider before applying ML models to new settings. Clinicians and decision-makers should take this into account when considering applying new ML models to their local settings.Copyright © 2022 The Author(s). Published by Wolters Kluwer Health, Inc. on behalf of the Society of Critical Care Medicine and Wolters Kluwer Health, Inc.

11 keywords...

hide…