• Joshua J Fenton, Hazel Tapp, Netra M Thakur, and Andrea L Pfeifle.
• From the Department of Family and Community Medicine and Center for Healthcare Policy and Research, University of California, Davis (JJF); Center for Primary Care Research, Department of Family Medicine, Atrium Health, Charlotte, NC (HT); University of North Carolina, School of Medicine, Advance Community Health Center, Raleigh (NMT); School of Medicine, Indiana University, Indianapolis (ALP). jjfenton@ucdavis.edu.
• J Am Board Fam Med. 2020 Nov 1; 33 (6): 986-991.

PurposeTo assess the reliability of peer review of abstracts submitted to academic family medicine meetings in North America.MethodsWe analyzed reviewer ratings of abstracts submitted: 1) as oral presentations to the North American Primary Care Research Group (NAPCRG) meeting from 2016 to 2019, as well as 2019 poster session or workshop submissions; and 2) in 12 categories to the Society of Teachers of Family Medicine (STFM) Spring 2018 meeting. In each category and year, we used a multi-level mixed model to estimate the abstract-level intraclass correlation coefficient (ICC) and the reliability of initial review (using the abstract-level ICC and the number of reviewers per abstract).ResultsWe analyzed review data for 1554 NAPCRG oral presentation abstracts, 418 NAPCRG poster or workshop abstracts, and 1145 STFM abstracts. Across all years, abstract-level ICCs for NAPCRG oral presentations were below 0.20 (range, 0.10 in 2019 to 0.18 in 2016) and were even lower for posters and workshops (range, 0.00-0.10). After accounting for the number of reviewers per abstract, reliabilities of initial review for NAPCRG oral presentations ranged from 0.24 in 2019 to 0.30 in 2016 and 0.00 to 0.18 for posters and workshops in 2019. Across 12 STFM submission categories, the median abstract-level ICC was 0.21 (range, 0.12-0.50) and the median reliability was 0.42 (range, 0.25-0.78).ConclusionsFor abstracts submitted to North American academic family medicine meetings, inter-reviewer agreement is often low, compromising initial review reliability. For many submission categories, program committees should supplement initial review with independent postreview assessments.© Copyright 2020 by the American Board of Family Medicine.

12 keywords...

hide…