• Byron C Wallace, Anna Noel-Storr, Iain J Marshall, Aaron M Cohen, Neil R Smalheiser, and James Thomas.
• College of Computer and Information Science, Northeastern University, Boston MA, USA.
• J Am Med Inform Assoc. 2017 Nov 1; 24 (6): 1165-1168.

ObjectivesIdentifying all published reports of randomized controlled trials (RCTs) is an important aim, but it requires extensive manual effort to separate RCTs from non-RCTs, even using current machine learning (ML) approaches. We aimed to make this process more efficient via a hybrid approach using both crowdsourcing and ML.MethodsWe trained a classifier to discriminate between citations that describe RCTs and those that do not. We then adopted a simple strategy of automatically excluding citations deemed very unlikely to be RCTs by the classifier and deferring to crowdworkers otherwise.ResultsCombining ML and crowdsourcing provides a highly sensitive RCT identification strategy (our estimates suggest 95%-99% recall) with substantially less effort (we observed a reduction of around 60%-80%) than relying on manual screening alone.ConclusionsHybrid crowd-ML strategies warrant further exploration for biomedical curation/annotation tasks.© The Author 2017. Published by Oxford University Press on behalf of the American Medical Informatics Association.

15 keywords...

hide…