• Laila Nasser, Shelley L McLeod, and Justin N Hall.
• Department of Emergency Medicine, Sunnybrook Health Sciences Centre, Department of Medicine, Temerty Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada; Department for Continuing Education, Oxford University, Oxford, England. Electronic address: laila.nasser@medportal.ca.
• Ann Emerg Med. 2024 Apr 1; 83 (4): 373379373-379.

Study ObjectiveThere is increasing interest in harnessing artificial intelligence to virtually triage patients seeking care. The objective was to examine the reliability of a virtual machine learning algorithm to remotely predict acuity scores for patients seeking emergency department (ED) care by applying the algorithm to retrospective ED data.MethodsThis was a retrospective review of adult patients conducted at an academic tertiary care ED (annual census 65,000) from January 2021 to August 2022. Data including ED visit date and time, patient age, sex, reason for visit, presenting complaint and patient-reported pain score were used by the machine learning algorithm to predict acuity scores. The algorithm was designed to up-triage high-risk complaints to promote safety for remote use. The predicted scores were then compared to nurse-led triage scores previously derived in real time using the electronic Canadian Triage and Acuity Scale (eCTAS), an electronic triage decision-support tool used in the ED. Interrater reliability was estimated using kappa statistics with 95% confidence intervals (CIs).ResultsIn total, 21,469 unique ED patient encounters were included. Exact modal agreement was achieved for 10,396 (48.4%) patient encounters. Interrater reliability ranged from poor to fair, as estimated using unweighted kappa (0.18, 95% CI 0.17 to 0.19), linear-weighted kappa (0.25, 95% CI 0.24 to 0.26), and quadratic-weighted kappa (0.36, 95% CI 0.35 to 0.37) statistics. Using the nurse-led eCTAS score as the reference, the machine learning algorithm overtriaged 9,897 (46.1%) and undertriaged 1,176 (5.5%) cases. Some of the presenting complaints under-triaged were conditions generally requiring further probing to delineate their nature, including abnormal lab/imaging results, visual disturbance, and fever.ConclusionThis machine learning algorithm needs further refinement before being safely implemented for patient use.Copyright © 2023 American College of Emergency Physicians. Published by Elsevier Inc. All rights reserved.

20 keywords...

hide…