• Surgery · Sep 2016

    Simulator and 2 tools: Validation of performance measures from a novel neurosurgery simulation model using the current Standards framework.

    • Deborah M Rooney, Bruce L Tai, Oren Sagher, Albert J Shih, David Andrew Wilkinson, and Luis E Savastano.
    • Department of Learning Health Sciences, University of Michigan Medical School, Ann Arbor, MI. Electronic address: dmrooney@med.umich.edu.
    • Surgery. 2016 Sep 1; 160 (3): 571-9.

    BackgroundVentriculostomy is a common neurosurgical procedure with a relatively steep learning curve. A low-cost, high-fidelity simulator paired with procedure-specific performance measures would provide a safe environment to teach ventriculostomy procedural skills. The same validated simulation model could also allow for assessment of trainees' proficiencies with measures that align with Accreditation Council for Graduate Medical Education milestones. This study extends previous work to evaluate validity evidence from the simulator, its newly developed performance assessment, the Ventricolostomy Procedural Assessment Tool, and the Objective Structured Assessment for Technical Skills.MethodsAfter Institutional Review Board exemption, performance data were collected from 11 novice and 3 expert neurosurgeons (n = 14). Participants self-reported their ability to perform tasks on the simulator using the Ventricolostomy Procedural Assessment Tool, an 11-item, step-wise instrument with 5-point rating scales ranging from 1 (unable to perform) to 5 (performs easily and smoothly). De-identified operative performances were videotaped and independently rated by 3 neurosurgeons, using the Ventricolostomy Procedural Assessment Tool and Objective Structured Assessment for Technical Skills. We evaluated multiple sources of validity evidence (2014 Standards) to examine psychometric quality of the measures and to test our assumption that the tools could discriminate between novice and expert performances adequately. We used a multifacet Rasch model and traditional indices, such as Cronbach alpha, intraclass correlation, and Wilcoxon signed-rank test estimates.ResultsValidity evidence relevant to test content and response processes was supported adequately. Evidence of internal structure was supported by high interitem consistency (n = 0.95) and inter-rater agreement for most Ventricolostomy Procedural Assessment Tool items (Intraclass correlation coefficient = [0.00, 0.91]) and all Objective Structured Assessment for Technical Skills items (Intraclass correlation coefficient = [0.80, 0.93]). Overall, novices performed at a lower level than experts on both scales (P < .05), supporting evidence relevant to relationships to other variables. Deeper analysis of novice/expert ratings indicated novices attained lower performances ratings for all Ventricolostomy Procedural Assessment Tool and Objective Structured Assessment for Technical Skills items, but statistical significance was only achieved for the Objective Structured Assessment for Technical Skills items (P < .01). Rater bias estimates were favorable, supporting evidence relevant to consequences of testing.ConclusionDespite a small sample, favorable evidence using current Standards supports the use of the novel simulator and both tools combined for skills training and performance assessment, but challenges (potential threats to validity) should be considered prior to implementation.Copyright © 2016 Elsevier Inc. All rights reserved.

      Pubmed     Full text   Copy Citation     Plaintext  

      Add institutional full text...

    Notes

     
    Knowledge, pearl, summary or comment to share?
    300 characters remaining
    help        
    You can also include formatting, links, images and footnotes in your notes
    • Simple formatting can be added to notes, such as *italics*, _underline_ or **bold**.
    • Superscript can be denoted by <sup>text</sup> and subscript <sub>text</sub>.
    • Numbered or bulleted lists can be created using either numbered lines 1. 2. 3., hyphens - or asterisks *.
    • Links can be included with: [my link to pubmed](http://pubmed.com)
    • Images can be included with: ![alt text](https://bestmedicaljournal.com/study_graph.jpg "Image Title Text")
    • For footnotes use [^1](This is a footnote.) inline.
    • Or use an inline reference [^1] to refer to a longer footnote elseweher in the document [^1]: This is a long footnote..

    hide…