• Jesse D Troy, Megan L Neely, Gina-Maria Pomann, Steven C Grambow, and Gregory P Samsa.
• Department of Biostatistics and Bioinformatics, Duke University School of Medicine, Durham, North Carolina, USA. Electronic address: Jesse.Troy@Duke.edu.
• J Pain Symptom Manage. 2023 Dec 1; 66 (6): e672e686e672-e686.

ContextThe CONSORT guideline defines a pilot trial as a small-scale version of a desired future efficacy trial that is intended to answer the key questions of whether and how a larger study should be done. For example, a pilot trial might evaluate different approaches to data collection or outcome measurement. However, pilot trials are unreliable for assessing treatment efficacy due to the statistical phenomenon called sampling variability.ObjectivesIn this tutorial we use computer simulation to demonstrate the influence of sampling variability on efficacy estimates from pilot trials, illustrating why pilot trial designs should not be used to evaluate whether a treatment is promising or not.MethodsWe simulate a 2-arm parallel group trial (N=20 per group) with a survival outcome as an example. Simulations are done under two scenarios: 1) the treatment is efficacious at the level of a hypothetical minimum clinically important difference (hazard ratio [HR] = 0.75); and 2) the treatment is not efficacious (HR=1).ResultsAs expected, in both simulated scenarios the range of observed results is distributed around the true treatment effect, HR=0.75 or HR=1. Importantly, ∼20% of trials simulated under scenario 1 incorrectly suggest the treatment may be harmful (HR > 1). Under scenario 2, half of the simulated studies incorrectly suggest the treatment is beneficial.ConclusionTreatment effect estimates from pilot trials should not be used to make future development decisions regarding a novel therapy because of the high risk of misleading conclusions.Copyright © 2023 The Authors. Published by Elsevier Inc. All rights reserved.

8 keywords...

hide…