• Mitchell G Goldenberg and Teodor P Grantcharov.
• International Centre for Surgical Safety, Keenan Centre for Biomedical Science, St. Michael's Hospital, Toronto, Ontario, Canada.
• Ann. Surg. 2019 Jan 1; 269 (1): 79-82.

ObjectiveTo describe a novel, outcome-based method of standard setting that differentiates between clinical outcomes rather than arbitrary educational goals.BackgroundStandard setting methods used in assessments of procedural skill are currently not evidence-driven or outcome-based. This represents a potential obstacle for the broad implementation of these evaluations in summative assessments such as certification and credentialing.MethodsThe concept is based on deriving a receiver operating characteristic curve from a regression model that incorporates measures of intraoperative surgeon performance and confounding patient characteristics. This allows the creation of a performance standard that best predicts a clinically significant outcome of interest. The discovery cohort used to create the predictive model was derived from pilot data that used the Global Evaluative Assessment of Robotic Skill assessment tool to predict patient urinary continence 3 months following robotic-assisted radical prostatectomy.ResultsA receiver operating characteristic curve with an area under the curve of 0.75 was created from predicted probability statistic generated by the predictive model. We chose a predicted probability of 0.35, based on an optimal tradeoff in sensitivity and specificity (Youden Index). Rearranging the regression equation, we determined the performance score required to predict a 35%, patient-adjusted probability of postoperative urinary incontinence.ConclusionsThis novel methodology is context, patient, and assessment-specific. Current standard setting methods do not account for the heterogeneity of the clinical environment. Workplace-based assessments in competency-based medical education require standards that are credible to the educator and the trainee. High-stakes assessments must ensure that surgeons have been evaluated to a standard that prioritizes satisfactory patient outcomes and safety.

24 keywords...

hide…