• C Jessica Dine, Judy A Shea, Caitlin B Clancy, Janae K Heath, William Pluta, and Jennifer R Kogan.
• Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA.
• J Gen Intern Med. 2024 Aug 21.

BackgroundInstitutions rely on student evaluations of teaching (SET) to ascertain teaching quality. Manual review of narrative comments can identify faculty with teaching concerns but can be resource and time-intensive.AimTo determine if natural language processing (NLP) of SET comments completed by learners on clinical rotations can identify teaching quality concerns.Setting And ParticipantsSingle institution retrospective cohort analysis of SET (n = 11,850) from clinical rotations between July 1, 2017, and June 30, 2018.Program DescriptionThe performance of three NLP dictionaries created by the research team was compared to an off-the-shelf Sentiment Dictionary.Program EvaluationThe Expert Dictionary had an accuracy of 0.90, a precision of 0.62, and a recall of 0.50. The Qualifier Dictionary had lower accuracy (0.65) and precision (0.16) but similar recall (0.67). The Text Mining Dictionary had an accuracy of 0.78 and a recall of 0.24. The Sentiment plus Qualifier Dictionary had good accuracy (0.86) and recall (0.77) with a precision of 0.37.DiscussionNLP methods can identify teaching quality concerns with good accuracy and reasonable recall, but relatively low precision. An existing, free, NLP sentiment analysis dictionary can perform nearly as well as dictionaries requiring expert coding or manual creation.© 2024. The Author(s).

3 keywords...

hide…