• Leo Nunnink, Carole Foot, Bala Venkatesh, Charlie Corke, Manoj Saxena, Mark Lucey, and Mark Jones.
• University of Queensland, Brisbane, QLD, Australia. l.nunnink@uq.edu.au.
• Crit Care Resusc. 2014 Mar 1; 16 (1): 6-12.

ObjectiveTo evaluate the use of high-fidelity simulation for summative high-stakes assessment of intensive care trainees, focusing on non-technical skills (NTS), testing feasibility and acceptability of simulation assessment, and the reliability of two NTS rating scales.Design, Setting And ParticipantsProspective observational study of senior intensive care trainees in a simulated specialist examination.MethodsParticipants undertook a simulated patient management scenario and were assessed using two rating scales: the Anaesthesia Non-technical Skills (ANTS) scale and the Ottawa Global Rating Scale (GRS). Assessors were trained, currently active, high-stakes examiners. Participants also completed a survey on simulation-based summative assessment.Outcome MeasuresThe inter-rater reliability of two rating scales for NTS assessment. We evaluated the feasibility of simulation-based assessment, and used survey results to assess acceptability to participants.ResultsSimulation assessment was feasible. Participants considered simulation-based high-stakes assessment to be acceptable and felt their scenario performance was reflective of real-world performance. Participants identified a need for debriefing following scenario-based assessment. Inter-rater reliability was fair for the ANTS and Ottawa GRS scores (intra-class correlation coefficient, 0.39 and 0.42, respectively). There was only fair agreement between raters for an NTS pass or fail (weighted kappa, 0.32) and for a technical skills pass or fail (weighted kappa, 0.36).ConclusionsSummative high-stakes assessment using a single simulated scenario was feasible and acceptable to senior intensive care trainees. The low inter-rater reliability for the ANTS and Ottawa GRS rating scales and for pass or fail discrimination may limit its incorporation into an existing examination format.

19 keywords...

hide…