• Steven Cogill, Kent Heberer, Amit Kaushal, Daniel Fang, and Jennifer Lee.
• VA Palo Alto Cooperative Studies Program Coordinating Center, Palo Alto, CA, USA.
• J Gen Intern Med. 2025 Jan 6.

BackgroundAdvances in artificial intelligence and machine learning have facilitated the creation of mortality prediction models which are increasingly used to assess quality of care and inform clinical practice. One open question is whether a hospital should utilize a mortality model trained from a diverse nationwide dataset or use a model developed primarily from their local hospital data.ObjectiveTo compare performance of a single-hospital, 30-day all-cause mortality model against an established national benchmark on the task of mortality prediction.Design/ParticipantsWe developed a single-hospital mortality prediction model using 9975 consecutive inpatient admissions at the Department of Veterans Affairs Palo Alto Healthcare System from July 26, 2018, to September 30, 2021, and compared performance against an established national model with similar features.Main MeasuresBoth the single-hospital model and the national model placed each patient in one of five prediction bins: < 2.5%, 2.5-5%, 5-10%, 10-30%, and ≥ 30% risks of 30-day mortality. Evaluation metrics included receiver operator characteristic area under the curve (ROC AUC), sensitivity, specificity, and balanced accuracy. Final comparisons were made between the single-hospital model trained on the full training set and the national model for both metrics and prediction overlap.Key ResultsWith sufficiently large training sets of 2720 or greater inpatient admissions, there was no statistically significant difference between the performances of the national model (ROC AUC 0.89, 95%CI [0.858, 0.919]) and single-hospital model (ROC AUC 0.878, 95%CI [0.84, 0.912]). For the 89 mortality events in the test set, the single-hospital model agreed with the national model risk assessment or an adjacent risk assessment in 92.1% of the encounters.ConclusionsA single-hospital inpatient mortality prediction model can achieve performance comparable to a national model when evaluated on a single-hospital population, given sufficient sample size.© 2024. This is a U.S. Government work and not under copyright protection in the US; foreign copyright protection may apply.

5 keywords...

hide…