-
- C Melville, M Rees, D Brookfield, and J Anderson.
- Department of Paediatrics, Keele University, Keele, UK. mea02@keele.ac.uk
- Med Educ. 2004 Oct 1;38(10):1117-25.
IntroductionIn 1997 the Royal College of Paediatrics and Child Health introduced portfolios to guide and monitor the learning of specialist registrars. We studied their value for assessment.MethodsUsing Bigg's SOLO criteria we devised a marking scheme based on 6 domains of competence: clinical, communication, teaching and learning, ethics and attitudes, management and evaluation, and creation of evidence. We rated portfolios according to quality of evidence presented and expectations by year of training. We similarly assessed trainee performance in the annual record of in-training assessment (RITA) interview. Specific advice based on the results of the first portfolio assessments was circulated to all trainees, instructing them to increase the structure and decrease the bulk of portfolios. A second sample of portfolios was reviewed a year later, using similar evaluations, to determine the effects.ResultsA total of 76 portfolios were assessed in year 1 by a single rater; 30 portfolios were assessed in year 2 by 2 independent raters. The quality of documentation improved from year 1 to year 2 but there was no significant increase in portfolio scores. The inter-rater correlation coefficient of the portfolio assessment method was 0.52 (Cohen's kappa 0.35). The inter-rater correlation coefficient of the RITA interview was 0.71 (Cohen's kappa 0.38). There was moderate inter-assessment correlation between portfolios and RITA interviews (kappa 0.26 in year 1 and 0.29 in year 2). Generalisability analysis suggested that 5 successive ratings by a single observer or independent ratings by 4 observers on the same occasion would be needed to yield a generalisability coefficient > 0.8 for overall portfolio rating.ConclusionsThis method of portfolio assessment is insufficiently reliable as a sole method for high stakes, single-instance assessment, but has a place as part of a triangulation process. Repeated portfolio assessment by paired observers would increase reliability. Longer term studies are required to establish whether portfolio assessment positively influences learner behaviour.
Notes
Knowledge, pearl, summary or comment to share?You can also include formatting, links, images and footnotes in your notes
- Simple formatting can be added to notes, such as
*italics*
,_underline_
or**bold**
. - Superscript can be denoted by
<sup>text</sup>
and subscript<sub>text</sub>
. - Numbered or bulleted lists can be created using either numbered lines
1. 2. 3.
, hyphens-
or asterisks*
. - Links can be included with:
[my link to pubmed](http://pubmed.com)
- Images can be included with:
![alt text](https://bestmedicaljournal.com/study_graph.jpg "Image Title Text")
- For footnotes use
[^1](This is a footnote.)
inline. - Or use an inline reference
[^1]
to refer to a longer footnote elseweher in the document[^1]: This is a long footnote.
.