• Hamed Akhlaghi, Sam Freeman, Cynthia Vari, Bede McKenna, George Braitberg, Jonathan Karro, and Bahman Tahayori.
• Department of Emergency Medicine, St Vincent's Hospital Melbourne, Melbourne, Victoria, Australia.
• Emerg Med Australas. 2024 Feb 1; 36 (1): 118124118-124.

ObjectiveArtificial intelligence (AI) has gradually found its way into healthcare, and its future integration into clinical practice is inevitable. In the present study, we evaluate the accuracy of a novel AI algorithm designed to predict admission based on a triage note after clinical implementation. This is the first of such studies to investigate real-time AI performance in the emergency setting.MethodsThe novel AI algorithm that predicts admission using a triage note was translated into clinical practice and integrated within St Vincent's Hospital Melbourne's electronic emergency patient management system. The data were collected from 1 January 2021 to 17 August 2022 to evaluate the diagnostic accuracy of the AI system after implementation.ResultsA total of 77 125 ED presentations were included. The live AI algorithm has a sensitivity of 73.1% (95% confidence interval 72.5-73.8), specificity of 74.3% (73.9-74.7), positive predictive value of 50% (49.6-50.4) and negative predictive value of 88.7% (88.5-89) with a total accuracy of 74% (73.7-74.3). The accuracy of the system was at the lowest for admission to psychiatric units (34%) and at the highest for gastroenterology and medical admission (84% and 80%, respectively).ConclusionOur study showed the diagnostic evaluation of a real-time AI clinical decision-support tool became less accurate than the original. Although real-time sensitivity and specificity of the AI tool was still acceptable as a decision-support tool in the ED, we propose that continuous training and evaluation of AI-enabled clinical support tools in healthcare are conducted to ensure consistent accuracy and performance to prevent inadvertent consequences.© 2023 The Authors. Emergency Medicine Australasia published by John Wiley & Sons Australia, Ltd on behalf of Australasian College for Emergency Medicine.

10 keywords...

hide…