• Thomas R O'Neill, Keith Stelter, and Ting Wang.
• From the American Board of Family Medicine, Lexington, KY (TRO, KS, TW); University of Minnesota - Mankato Family Medicine Residency Program, Mankato, MN (KS). toneill@theabfm.org.
• J Am Board Fam Med. 2025 Jan 30.

PurposeTo determine whether the construct of family medicine clinical decision making ability was invariant across modes of administration, the 1-day examination and the longitudinal assessment. We attempted to identify item characteristics associated with differences in difficulty across modes of administration.MethodsThe data were item difficulty calibrations based on examinee responses to the 1-day examination and the longitudinal assessment. A repeated measures design was employed to identify question calibration differences across modes of administration, so that the stability of the question difficulty across modes of administration could be assessed. A qualitative review of the flagged questions was conducted to identify characteristics associated with questions becoming easier or more difficult.ResultsThe correlation between the pairs of calibrations was moderately positive r(298) = 0.558, P < .001 suggesting that the questions are functioning somewhat similarly across the different modes of administration; however, the scatterplot demonstrates that many of the questions became easier. Of the 298 repeated measures t test, 37% (110) did not show a significant difference, 43% (128) became easier on the longitudinal assessment, and 20% (60) became more difficult.ConclusionsThis study suggests that changes in item difficulty do occur when extra time and the use of external resources are permitted. Usually the questions get easier, but in some cases the question becomes more difficult. Possible reasons for this are presented, and a method to adjust the item difficulty in a way to maintain a single construct is presented.© Copyright by the American Board of Family Medicine.

9 keywords...

hide…