Research synthesis methods
-
Comparative Study
Comparison of information sources used in Cochrane and non-Cochrane systematic reviews: A case study in the field of anesthesiology and pain.
It has been reported that information sources searched in systematic reviews (SRs) are insufficiently comprehensive. We analyzed information sources used in SRs, as well as how up-to-date were the searches. ⋯ SRs in the field of anesthesiology and pain often neglect to search all possible information sources, particularly in NCSRs. Cochrane reviews had more comprehensive searching and shorter search to publication time.
-
Machine learning (ML) algorithms have proven highly accurate for identifying Randomized Controlled Trials (RCTs) but are not used much in practice, in part because the best way to make use of the technology in a typical workflow is unclear. In this work, we evaluate ML models for RCT classification (support vector machines, convolutional neural networks, and ensemble approaches). We trained and optimized support vector machine and convolutional neural network models on the titles and abstracts of the Cochrane Crowd RCT set. ⋯ We demonstrate that ML approaches better discriminate between RCTs and non-RCTs than widely used traditional database search filters at all sensitivity levels; our best-performing model also achieved the best results to date for ML in this task (AUROC 0.987, 95% CI, 0.984-0.989). We provide practical guidance on the role of ML in (1) systematic reviews (high-sensitivity strategies) and (2) rapid reviews and clinical question answering (high-precision strategies) together with recommended probability cutoffs for each use case. Finally, we provide open-source software to enable these approaches to be used in practice.
-
Systematic reviews are essential to produce trustworthy guidelines. To assess the certainty of a body of evidence included in a systematic review the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group has developed an approach that is currently used by over 100 organisations, including the World Health Organization and the Cochrane Collaboration. ⋯ Summary statistical information and assessments of certainty are presented in GRADE evidence summary tables, which can be produced using GRADE's official GRADEpro software tool (www.gradepro.org/). The evidence summary tables feed into the GRADE Evidence to Decision frameworks which guideline panels can use to produce recommendations.
-
Conventional meta-analysis estimators are weighted means of study measures, meant to estimate an overall population measure. For measures such as means, mean differences and risk differences, a weighted arithmetic mean is the conventional estimator. When the measures are ratios, such as odds ratios, logarithms of the study measures are most frequently used, and the back-transform is a weighted geometric mean, rather than the arithmetic mean. ⋯ However, when the weights are the usual reciprocal variance estimates, the inequalities go in the opposite direction. The use of reciprocal variance weights is therefore questioned as perhaps having a fundamental flaw. An example is shown of a meta-analysis of frequencies of two classes of drug-resistant HIV-1 mutations.
-
Assessing the quality of included studies is a vital step in undertaking a systematic review. The recently revised Quality Assessment of Diagnostic Accuracy Studies (QUADAS) tool (QUADAS-2), which is the only validated quality assessment tool for diagnostic accuracy studies, does not include specific criteria for assessing comparative studies. As part of an assessment that included comparative diagnostic accuracy studies, we used a modified version of QUADAS-2 to assess study quality. ⋯ We have presented our modified version of QUADAS-2 and outlined some key issues for consideration when assessing the quality of comparative diagnostic accuracy studies, to help guide other systematic reviewers conducting comparative diagnostic reviews. Until QUADAS is updated to incorporate assessment of comparative studies, QUADAS-2 can be used, although modification and careful thought is required. It is important to reflect upon whether aspects of study design and methodology favour one of the tests over another.