• Borislav D Dimitrov, Nicola Motterlini, and Tom Fahey.
• Academic Unit of Primary Care and Population Sciences, University of Southampton, Southampton, United Kingdom ; HRB Centre for Primary Care Research, Department of General Medicine, Division of Population Health Sciences, Royal College of Surgeons in Ireland, Dublin, Ireland.
• Clin Epidemiol. 2015 Jan 1;7:267-80.

ObjectiveEstimating calibration performance of clinical prediction rules (CPRs) in systematic reviews of validation studies is not possible when predicted values are neither published nor accessible or sufficient or no individual participant or patient data are available. Our aims were to describe a simplified approach for outcomes prediction and calibration assessment and evaluate its functionality and validity.Study Design And MethodsMethodological study of systematic reviews of validation studies of CPRs: a) ABCD(2) rule for prediction of 7 day stroke; and b) CRB-65 rule for prediction of 30 day mortality. Predicted outcomes in a sample validation study were computed by CPR distribution patterns ("derivation model"). As confirmation, a logistic regression model (with derivation study coefficients) was applied to CPR-based dummy variables in the validation study. Meta-analysis of validation studies provided pooled estimates of "predicted:observed" risk ratios (RRs), 95% confidence intervals (CIs), and indexes of heterogeneity (I(2) ) on forest plots (fixed and random effects models), with and without adjustment of intercepts. The above approach was also applied to the CRB-65 rule.ResultsOur simplified method, applied to ABCD(2) rule in three risk strata (low, 0-3; intermediate, 4-5; high, 6-7 points), indicated that predictions are identical to those computed by univariate, CPR-based logistic regression model. Discrimination was good (c-statistics =0.61-0.82), however, calibration in some studies was low. In such cases with miscalibration, the under-prediction (RRs =0.73-0.91, 95% CIs 0.41-1.48) could be further corrected by intercept adjustment to account for incidence differences. An improvement of both heterogeneities and P-values (Hosmer-Lemeshow goodness-of-fit test) was observed. Better calibration and improved pooled RRs (0.90-1.06), with narrower 95% CIs (0.57-1.41) were achieved.ConclusionOur results have an immediate clinical implication in situations when predicted outcomes in CPR validation studies are lacking or deficient by describing how such predictions can be obtained by everyone using the derivation study alone, without any need for highly specialized knowledge or sophisticated statistics.

5 keywords...

hide…