• M J Roberts, T C E Gale, P J A Sice, and I R Anderson.
• Collaboration for the Advancement of Medical Education Research and Assessment, Plymouth University Peninsula Schools of Medicine and Dentistry, Plymouth, UK. martin.roberts@pms.ac.uk
• Anaesthesia. 2013 Jun 1;68(6):591-9.

AbstractSelection to specialty training is a high-stakes assessment demanding valuable consultant time. In one initial entry level and two higher level anaesthesia selection centres, we investigated the feasibility of using staff participating in simulation scenarios, rather than observing consultants, to rate candidate performance. We compared participant and observer scores using four different outcomes: inter-rater reliability; score distributions; correlation of candidate rankings; and percentage of candidates whose selection might be affected by substituting participants' for observers' ratings. Inter-rater reliability between observers was good (correlation coefficient 0.73-0.96) but lower between participants (correlation coefficient 0.39-0.92), particularly at higher level where participants also rated candidates more favourably than did observers. Station rank orderings were strongly correlated between the rater groups at entry level (rho 0.81, p < 0.001) but weaker at the two higher level centres (rho 0.52, p = 0.018; rho 0.58, p = 0.001). Substituting participants' for observers' ratings had less effect once scores were combined with those from other selection centre stations. Selection decisions for 0-20% of candidates could have changed, depending on the numbers of training posts available. We conclude that using participating raters is feasible at initial entry level only.Anaesthesia © 2013 The Association of Anaesthetists of Great Britain and Ireland.

13 keywords...

hide…