EBM bibliography updates – May 2024

If you haven’t seen it yet, I have a collection of important evidence based medicine articles that will really help you understand the science behind your medical practice. I constantly update it with new papers I read, and occasionally publish summaries of those updates. This update contains a lot about non-inferiority trials, which is probably not surprising considering this recent very long post about the many problems with non-inferiority trial design. 

Aberegg SK, Hersh AM, Samore MH. Empirical Consequences of Current Recommendations for the Design and Interpretation of Noninferiority Trials. J Gen Intern Med. 2018 Jan;33(1):88-96. doi: 10.1007/s11606-017-4161-4. Epub 2017 Sep 5. PMID: 28875400

• A review of 182 noninferiority trials in top rated journals found numerous problems, including the fact that about 12% of the time the experimental therapy was statistically worse than active control, but the CONSORT recommended conclusion for the trial was “noninferior”. (Aberegg 2018)
• This same study finds that an astonishing 77% of published non-inferiority trials make the claim of non-inferiority or superiority, as compared to only 2% that conclude that the novel therapy is inferior. If non-inferiority trials essentially never conclude that a treatment is inferior, that sounds a lot like there is significant bias, or there is a fundamental flaw in this trial design. (Prasad 2017)

Flacco ME, Manzoli L, Boccia S, Capasso L, Aleksovska K, Rosso A, Scaioli G, De Vito C, Siliquini R, Villari P, Ioannidis JP. Head-to-head randomized trials are mostly industry sponsored and almost always favor the industry sponsor. J Clin Epidemiol. 2015 Jul;68(7):811-20. doi: 10.1016/j.jclinepi.2014.12.016. Epub 2015 Feb 7. PMID: 25748073

• In head to head RCTs with industry funding, industry funding is strongly associated with a favourable outcome for the sponsor
• In non-inferiority/equivalence designs, 97% of trials reported favourable outcomes for the sponsor of the trial. 

Le Henanff A, Giraudeau B, Baron G, Ravaud P. Quality of reporting of noninferiority and equivalence randomized trials. JAMA. 2006 Mar 8;295(10):1147-51. doi: 10.1001/jama.295.10.1147. PMID: 16522835

• A review of 162 noninferiority and equivalence trials found significant deviations from accepted good research practice
  • 80% of trials did not provide a justification for the non inferiority margin being used
  • 28% did not account for the non-inferiorty margin in the sample size calculation

Ofori S, Cafaro T, Devereaux PJ, Marcucci M, Mbuagbaw L, Thabane L, Guyatt G. Noninferiority margins exceed superiority effect estimates for mortality in cardiovascular trials in high-impact journals. J Clin Epidemiol. 2023 Sep;161:20-27. doi: 10.1016/j.jclinepi.2023.06.022. Epub 2023 Jul 6. PMID: 37421996

• This is a review of RCTs from the cardiovascular literature
• The non-inferiority margins chosen are often larger than the benefits we look for in superiority trials. “Over 70% of noninferiority trials chose margins in mortality outcomes greater than the median of 2.1% that superiority trials considered important. In other words, although 50% of superiority trial authors considered an important effect 2.1% or more (and 50% effects even smaller), 71.8% of noninferiority trials considered this loss of benefit acceptable.”
• In other words, an superiority RCT might demonstrate a 2.1% absolute reduction in mortality, and we would all agree that is a benefit for patients. And then a non-inferiority trial of the exact same 2 interventions might also show a 2.1% absolute difference, but then conclude that the treatment is “non-interior”.
• From the mouth of Gordon Guyatt himself (on Twittter): “Clinicians should ignore these margins & statistical claims of non-inferiority & instead think how their patients would balance benefits/harms/burdens.” (If that is the case, I wonder about the real value of this trial design? I think it should probably be largely abandoned.)

van Zwet E, Gelman A, Greenland S, Imbens G, Schwab S, Goodman SN. A New Look at P Values for Randomized Clinical Trials. NEJM Evid. 2024 Jan;3(1):EVIDoa2300003. doi: 10.1056/EVIDoa2300003. Epub 2023 Dec 22. Erratum in: NEJM Evid. 2024 Feb;3(2):EVIDx2400007. PMID: 38320512

• In a surprise to no one, medical trials are almost always way too small. (Designed for speed, cost, or ease rather than based on realistic power calculations.)
• These authors looked at the primary outcome of 23,551 RCTs. The median power of the trials was only 13%, and only 13% of all trials reached 80%, despite essentially all trials containing a power calculation that claims to target that level. 
• As a result, there is a high probability that trials with statistically significant results over-estimate the magnitude of those results. “We estimate that there is a 75% probability that the magnitude of the effect is overestimated by at least 5%, a 50% probability that it is overestimated by at least 56%, and a 25% probability that it is overestimated by at least 181%. This phenomenon is like the infamous “winner’s curse” in auctions..”
• Based on their math, a trial with a p value less than 0.05 has only a 37% chance of being replicated!
• Underpowered trials also mean that we will see a lot of ‘not statistically significant’ results for real, important effects. In other words, small trials hurt us in both directions.

Kotani Y, Turi S, Ortalda A, Baiardo Redaelli M, Marchetti C, Landoni G, Bellomo R. Positive single-center randomized trials and subsequent multicenter randomized trials in critically ill patients: a systematic review. Crit Care. 2023 Nov 28;27(1):465. doi: 10.1186/s13054-023-04755-5. PMID: 38017475

• We have a false positive problem in medicine
• This systematic review, they looked for any single center RCT published in New England Journal of Medicine, JAMA, or Lancet that demonstrated a mortality benefit in critically ill patients, and asked the simple question: what did follow-up multicenter trials show?
  • Of the 19 single center RCTs that demonstrated a mortality benefit, only 16 had a follow-up multicenter RCT, and only one of those follow-up RCTs confirmed the original findings, with another follow-up multicenter RCT actually showing the exact opposite, with an increase in mortality. 
• Approaching new findings in medical research with extreme caution is not cynical, its good science.