• Thomas J Beckman, David A Cook, and Jayawant N Mandrekar.
• Division of General Internal Medicine, Department of Internal Medicine, Mayo Clinic College of Medicine, Mayo Clinic and Mayo Foundation, Rochester, MN, USA. Beckman.Thomas@mayo.edu
• J Gen Intern Med. 2005 Dec 1; 20 (12): 115911641159-64.

BackgroundAlthough a variety of validity evidence should be utilized when evaluating assessment tools, a review of teaching assessments suggested that authors pursue a limited range of validity evidence.ObjectivesTo develop a method for rating validity evidence and to quantify the evidence supporting scores from existing clinical teaching assessment instruments.DesignA comprehensive search yielded 22 articles on clinical teaching assessments. Using standards outlined by the American Psychological and Education Research Associations, we developed a method for rating the 5 categories of validity evidence reported in each article. We then quantified the validity evidence by summing the ratings for each category. We also calculated weighted kappa coefficients to determine interrater reliabilities for each category of validity evidence.Main ResultsContent and Internal Structure evidence received the highest ratings (27 and 32, respectively, of 44 possible). Relation to Other Variables, Consequences, and Response Process received the lowest ratings (9, 2, and 2, respectively). Interrater reliability was good for Content, Internal Structure, and Relation to Other Variables (kappa range 0.52 to 0.96, all P values < .01), but poor for Consequences and Response Process.ConclusionsContent and Internal Structure evidence is well represented among published assessments of clinical teaching. Evidence for Relation to Other Variables, Consequences, and Response Process receive little attention, and future research should emphasize these categories. The low interrater reliability for Response Process and Consequences likely reflects the scarcity of reported evidence. With further development, our method for rating the validity evidence should prove useful in various settings.

11 keywords...

hide…