• Brian C Ricci, Jonathan Sachs, Konrad Dobbertin, Faiza Khan, and David A Dorr.
• Department of Internal Medicine, Division of General Internal Medicine and Geriatrics, Oregon Health & Science University, School of Medicine, 3181 SW Sam Jackson Park Rd. Mail Code: L-475, Portland, OR, 97239, USA. riccib@ohsu.edu.
• J Gen Intern Med. 2022 Feb 1; 37 (3): 601607601-607.

BackgroundIn primary care risk stratification, automated algorithms do not consider the same factors as providers. The process of adjudication, in which providers review and adjust algorithm-derived risk scores, may improve the prediction of adverse outcomes.ObjectiveWe assessed the patient factors that influenced provider adjudication behavior and evaluated the performance of an adjudicated risk model against a commercial algorithm.Design(1) Structured interviews with primary care providers (PCP) and multivariable regression analysis and (2) receiver operating characteristic curves (ROC) with sensitivity analyses.ParticipantsPrimary care patients aged 18 years and older with an adjudicated risk score. APPROACH AND MAIN MEASURES: (1) Themes from structured interviews and discrete variables associated with provider adjudication behavior; (2) comparison of concordance statistics and sensitivities between risk models.Key Results47,940 patients were adjudicated by PCPs in 2018. Interviews revealed that, in adjudication, providers consider disease severity, presence of self-management skills, behavioral health, and whether a risk score is actionable. Provider up-scoring from the algorithmic risk score was significantly associated with patient male sex (OR 1.24, CI 1.15-1.34), age > 65 (OR 2.55, CI 2.24-2.91), Black race (1.26, CI 1.02-1.55), polypharmacy >10 medications (OR 4.87, CI 4.27-5.56), a positive depression screen (OR 1.57, CI 1.43-1.72), and hemoglobin A1c >9 (OR 1.89, CI 1.52-2.33). Overall, the adjudicated risk model performed better than the commercial algorithm for all outcomes: ED visits (c-statistic 0.689 vs. 0.684, p < 0.01), hospital admissions (c-statistic 0.663 vs. 0.649, p < 0.01), and death (c-statistic 0.753 vs. 0.721, p < 0.01). When limited to males or seniors, the adjudicated models displayed either improved or non-inferior performance compared to the commercial model.ConclusionsProvider adjudication of risk stratification improves model performance because providers have a personal understanding of their patients and are able to apply their training to clinical decision-making.© 2021. Society of General Internal Medicine.

15 keywords...

hide…