• Anneliese Arno, James Thomas, Byron Wallace, Iain J Marshall, Joanne E McKenzie, and Julian H Elliott.
• EPPI-Centre, University College London, London, United Kingdom, and School of Public Health and Preventive Medicine, Monash University, Melbourne, Victoria, Australia (A.A.).
• Ann. Intern. Med. 2022 Jul 1; 175 (7): 100110091001-1009.

BackgroundAutomation is a proposed solution for the increasing difficulty of maintaining up-to-date, high-quality health evidence. Evidence assessing the effectiveness of semiautomated data synthesis, such as risk-of-bias (RoB) assessments, is lacking.ObjectiveTo determine whether RobotReviewer-assisted RoB assessments are noninferior in accuracy and efficiency to assessments conducted with human effort only.DesignTwo-group, parallel, noninferiority, randomized trial. (Monash Research Office Project 11256).SettingHealth-focused systematic reviews using Covidence.ParticipantsSystematic reviewers, who had not previously used RobotReviewer, completing Cochrane RoB assessments between February 2018 and May 2020.InterventionIn the intervention group, reviewers received an RoB form prepopulated by RobotReviewer; in the comparison group, reviewers received a blank form. Studies were assigned in a 1:1 ratio via simple randomization to receive RobotReviewer assistance for either Reviewer 1 or Reviewer 2. Participants were blinded to study allocation before starting work on each RoB form.MeasurementsCo-primary outcomes were the accuracy of individual reviewer RoB assessments and the person-time required to complete individual assessments. Domain-level RoB accuracy was a secondary outcome.ResultsOf the 15 recruited review teams, 7 completed the trial (145 included studies). Integration of RobotReviewer resulted in noninferior overall RoB assessment accuracy (risk difference, -0.014 [95% CI, -0.093 to 0.065]; intervention group: 88.8% accurate assessments; control group: 90.2% accurate assessments). Data were inconclusive for the person-time outcome (RobotReviewer saved 1.40 minutes [CI, -5.20 to 2.41 minutes]).LimitationVariability in user behavior and a limited number of assessable reviews led to an imprecise estimate of the time outcome.ConclusionIn health-related systematic reviews, RoB assessments conducted with RobotReviewer assistance are noninferior in accuracy to those conducted without RobotReviewer assistance.Primary Funding SourceUniversity College London and Monash University.

13 keywords...

hide…