• J Graham, G Hocking, and E Giles.
• Department of Anaesthesia, Sir Charles Gairdner Hospital, Perth, WA 6009, Australia. jgraham@meddent.uwa.edu.au
• Br J Anaesth. 2010 Apr 1; 104 (4): 440445440-5.

BackgroundPerformance assessment is becoming increasingly necessary in the medical workplace. Hospitals and patients expect safety, and under-performance by a doctor can compromise standards. By describing and quantifying performance, positive behaviour can be encouraged and unsafe behaviour remedied. Anaesthesia Non-Technical Skills (ANTS) is a behavioural marker system that can be used to assess non-technical skills in the workplace.MethodsWe determined whether specialist anaesthetists could be reliably trained to use ANTS at an assessor level in an 8 h programme. Unscripted videos of routine anaesthesia were produced for training and assessment purposes. Twenty-six participants attended rater training. Exercises in behaviour observation, rater error, frame of reference and performance dimension, and the use of ANTS were conducted throughout the day. Five videos were selected for formal assessment and data collected. Intra-class correlations (ICCs) were calculated for each element.ResultsThe accepted value of r>0.7 was not reached. ICC calculated for each element was 0.11-0.62. Comparison of participants scores with those of expert raters showed poor agreement.ConclusionsAnaesthetists could not be trained to reliably use ANTS as a summative assessment tool using our 1 day programme. There was an inadequate correlation of scores between participants and experts. Two major problems contributed to the lack of agreement. Observed behaviours were often misclassified into the incorrect element and safety beliefs varied among anaesthetists. Other reasons for the failure to achieve success and potential future direction are discussed.

This article appears in the collection: Non-technical qualities of anesthesiology.

17 keywords...

hide…