• L Grosso and J Norcini.
• American Board of Internal Medicine, Philadelphia, PA 19104.
• Acad Med. 1994 Dec 1;69(12):979-82.

BackgroundAs part of the admission process to the American Board of Internal Medicine's certifying examination in internal medicine, training program directors evaluate residents in several components of clinical competence, including medical knowledge. Research suggested that these ratings had different meanings across programs. A report comparing certifying examination performance and ratings of medical knowledge at the program and national levels was developed and sent to program directors after the 1988 through 1992 examinations. The present study investigated whether feedback helped program directors identify where their residents ranked nationally.MethodSubjects were first-time takers of the 1986 through 1992 certifying examinations in internal medicine who took the examination in the year they completed training and who received ratings of 4 through 9 in medical knowledge. All subjects were from programs contributing examinees in all seven study years and that received feedback in 1988 through 1991. Year-by-year distributions of program mean percentages of examinees receiving each rating of medical knowledge (4 through 9) were generated. Program means for equated examination scores and ratings of medical knowledge were computed for each year. Correlations between program mean scores and ratings were also computed.ResultsThe distributions of the ratings were stable across the study years. Mean scores declined while mean ratings were unchanged. At the same time, correlations between scores and ratings increased. The biggest one-year change was from 1989 to 1990 (.49 to .57).ConclusionSince equated scores are directly comparable, declining mean scores but unchanged mean ratings suggest that the standards applied by program directors drifted downward. The increasing correlations suggest that program directors improved in their abilities to evaluate residents relative to a common standard. It is not clear what effect the feedback had on program directors' evaluations. It is encouraging, however, to see a higher level of agreement among program directors on the meaning of the ratings.

14 keywords...

hide…