Morgenstern, J. Killer antibiotic stewardship strategy? (The ADAPT-Sepsis trial), First10EM, December 16, 2024. Available at:
https://doi.org/10.51684/FIRS.139764
Antibiotic resistance is an existential problem. Optimizing sepsis care, including the appropriate length of antibiotic therapy, is an important goal. So should we be willing to allow an increase in mortality to reduce total antibiotic use, as the ADAPT-Sepsis non-inferiority design seems to imply?
The paper
Dark P, Hossain A, McAuley DF, Brealey D, Carlson G, Clayton JC, Felton TW, Ghuman BK, Gordon AC, Hellyer TP, Lone NI, Manazar U, Richards G, McCullagh IJ, McMullan R, McNamee JJ, McNeil HC, Mouncey PR, Naisbitt MJ, Parker RJ, Poole RL, Rostron AJ, Singer M, Stevenson MD, Walsh TS, Welters ID, Whitehouse T, Whiteley S, Wilson P, Young KK, Perkins GD, Lall R; ADAPT-Sepsis Collaborators. Biomarker-Guided Antibiotic Duration for Hospitalized Patients With Suspected Sepsis: The ADAPT-Sepsis Randomized Clinical Trial. JAMA. 2024 Dec 9. doi: 10.1001/jama.2024.26458. PMID: 39652885
The Methods
ADAPT-Sepsis is a 3 group multicentre RCT from 41 ICUs in the United Kingdom.
Patients
Adult patients in the ICU with suspected sepsis, with antibiotics started within 24 hours of enrollment, and expected to be continued for at least 3 days.
Exclusions: Need for a prolonged (>21 days) course of antibiotics, severely immunocompromised from a cause other than sepsis, expected to receive IL-6 receptor inhibitors (eg, tocilizumab or sarilumab), sepsis treatments likely to stop within 24 hours because of futility.
Intervention
Group 1: Procalcitonin guided antibiotic advice.
Group 2: CRP guided antibiotic advice.
CRP, but not PCT, could be measured outside of the study protocol if deemed necessary by the clinician but “could not be used for protocolized antibiotic duration guidance”, although that seems impossible to enforce.
Comparison
Standard care.
Outcome
The primary outcome was total antibiotic duration (days), from randomization to 28 days.
The primary safety outcome was 28 day all cause mortality.
The Results
Out of 16,109 patients screened, they ultimately include 2760 patients. Unfortunately, a large number of these exclusions were eligible patients, including 396 excluded with no reason given, and 148 excluded because the medical team declined to participate. The mean age was 60, 40% are female, and the groups appear well matched across all reported baseline characteristics.
PCT guided care resulted in a statistically significant decrease in antibiotic use (9.8 vs 10.7 days, mean difference 0.88 days, 95% CI 0.19 to 1.58; P = 0.01).
CRP guided care resulted in no difference in antibiotic use (10.6 vs 10.7 days, p=0.79).
They are able to conclude that the PCT guided treatment was “non-inferior” for all cause mortality, but that is based on a ridiculous non-inferiority margin, discussed more below. Mortality was 19.4% with standard care, 20.9% with procalcitonin, and 21.1% with CRP. As compared to standard care, procalcitonin was “non-inferior” (95% CI for mortality -2.18 to 5.32%) and CRP was “not non-inferior” (95% CI -2.07 to 5.45%).
My thoughts
Before we get into any issues, I think it is important to frame the results of this non-inferiority study correctly. Assuming you aren’t worried about any bias in this study, what it says is that a procalcitonin guided strategy decreases antibiotic use by an average of less than 1 day, while resulting in no more than a 5.39% increase in mortality. That is how the study was designed. Although no one really understands non-inferiority trials, that is what this trial was designed to show. Without even looking at the results, would you accept an antibiotic stewardship strategy that resulted in up to a 5.4% increase in mortality? Would you accept any increase in mortality?
It is truly unfortunate when a trial’s fatal flaw is so obvious from the methodology. This should have been caught before the trial was run. There is no way clinicians should ever accept a 5.4% non-inferiority margin for mortality. (In fact, it isn’t clear we should ever use a non-inferiority design to look at mortality, because there is almost no benefit that would allow us to accept an increase in mortality.) Before I even read the results section, it was clear that this trial could not be practice changing. This is why peer review needs to happen before clinical data is collected, not after completion of the trial.
Although the major issue with this trial centers around non-inferiority and the mortality outcome, it is worth noting that I don’t even think their primary outcome of an antibiotic reduction is trustworthy. Based on my read, this trial was heavily biased towards finding a reduction in antibiotic use. You can see from the table above that the advice given in the group of patients with high procalcitonin and high CRP was the exact same advice given to clinicians in the standard care group. However, when the only options available are a message of “stop antibiotics” or to continue with “standard care”, it is very easy to translate the “standard care” message into a suggestion that antibiotics should be considered. This is especially true because the trial was randomized at the individual patient level, so a single clinician could get used to seeing a message to stop antibiotics in procalcitonin trial patients, and therefore not stop antibiotics when that message was not displayed. (In other words, it would be very easy to mentally translate the message to continue standard care into the message that the patient still has a high PCT or CRP.)
Of course, the major problem with this trial was the chosen non-inferiority margin. They designed their trial with a 5.4% non-inferiority margin for mortality. That is a ridiculous choice. First of all, it is very hard to believe that this was their real apriori choice. Non-inferiority margins are chosen clinically, not calculated. No one chooses a number like 5.4%. Non-inferiority margins are essentially always whole numbers, because that is what a normal human being would choose. The use of the overly precise 5.4% reeks of p-hacking. In fact, if they had used the rational 5% that would have been used in basically any other non-inferiority study, the results of this study would have been “not non-inferior”. I obviously have no inside insight, but on its face it is almost impossible to believe that these results are not p-hacked. Note: Although it is very odd to have a decimal point in the clinically chosen non-inferiority margin, the 5.4% was clearly noted in a published 2017 protocol. It remains an inappropriate threshold, as discussed below, but there is clearly not any post-hoc adjustment to their statistical analysis. (Thank you to Jon Hansel for helping me find the information I had previously missed.)
More importantly, the 5.4% margin should have been rejected out of hand before the trial was even started. They designed the trial saying “we are happy with up to a 5.4% increase in mortality as long as patients are given a few fewer doses of antibiotics.” How does that get through initial review? How does that get through an IRB? No rational human being would accept a 5% increase in mortality to decrease antibiotics by 1 day. As designed, this trial should never have been run.
This is so mind blowing to me, I want to phrase this another way. Imagine you design a study, and you are asked what it will show in the best possible case, assuming the results are positive. For the study, the answer is, “we are hoping to show that our procalcitonin strategy will decrease antibiotic use, and that mortality will increase by no more than 5.4%”. That was their best case scenario. How does that study get funded? How does that study even get pitched?!
They were able to conclude “non-inferiority”, meaning we know that the difference in mortality is not more than 5.4%. However, non-inferiority is not equivalence, and the actual mortality was 19.4% in the standard care group and 20.9% in the PCT guided group. In other words, there was a 1.5% absolute increase in mortality, with only a putative benefit of 1 day less on antibiotics. The 95% confidence interval includes the possibility of a 5.3% increase in mortality. The best estimate from this trial is that using the described procalcitonin strategy will result in an extra 1.5 deaths per 100 patients, with a 95% confidence interval that includes an additional 5.3 deaths per hundred patients.
Ultimately, I am not sure how this procalcitonin strategy could result in a significant increase in mortality, and it is possible that they 1.5% increase in mortality is noise. However, for the time being, we have to assume the increase is real, because unfortunately the trial was designed with the intrinsic assumption that a 5.39% increase in mortality was acceptable.
I don’t know how JAMA published this trial as if the results are positive. These results are awful, and it is very clear that no one should be using a PCT guided strategy based on these results.
Bottom line
This RCT demonstrates that a procalcitonin guided strategy might decrease antibiotic use by about 1 day in ICU sepsis patients, but with the cost of up to a 5.4% increase in mortality. Obviously, this strategy should not be employed.
I’m getting very close to the 50 Patreon subscriber goal that would allow me to remove ads from First10EM. Thanks to all those who support the project!
Other FOAMed
You don’t understand non-inferiority trials (and neither do I)
PulmCrit: PulmCrit: ADAPT and SCREEN trials are full of sound and fury, signifying little
Evidence based medicine is easy
Evidence based medicine resources
References
Dark P, Hossain A, McAuley DF, Brealey D, Carlson G, Clayton JC, Felton TW, Ghuman BK, Gordon AC, Hellyer TP, Lone NI, Manazar U, Richards G, McCullagh IJ, McMullan R, McNamee JJ, McNeil HC, Mouncey PR, Naisbitt MJ, Parker RJ, Poole RL, Rostron AJ, Singer M, Stevenson MD, Walsh TS, Welters ID, Whitehouse T, Whiteley S, Wilson P, Young KK, Perkins GD, Lall R; ADAPT-Sepsis Collaborators. Biomarker-Guided Antibiotic Duration for Hospitalized Patients With Suspected Sepsis: The ADAPT-Sepsis Randomized Clinical Trial. JAMA. 2024 Dec 9. doi: 10.1001/jama.2024.26458. PMID: 39652885
6 thoughts on “Killer antibiotic stewardship strategy? (The ADAPT-Sepsis trial)”
Thank you, I enjoy and learn a lot about the way you break down and find the hacking in those large heavily-funded studies.
Is antibiotic usage in critical care medicine even an important concept for antibiotic starship and antimicrobial resistance? I would think that the global use of antibiotics is much higher in primary care and the general wards and having a strategy to find out who not to start them on int these scenarios would be much more valuable than reducing duration in critical care by < 1 day for a 1.5% absolute increase in mortality.
I totally agree with the weird choice for NI margin and the controversy with mortality in NIT.
But, as it was mentioned in twitter’s discussion by Todd Lee, because of the results of the very recent and even with a larger sample BALANCE trial (7 vs 14 days – https://www.nejm.org/doi/full/10.1056/NEJMoa2404991), it doesn’t make much sense that the mortality increase is real and not just statistical noise (it would imply 10 days would be better than 7 days)
Any chance you can link me? Can you elaborate on what you mean towards the end there with the 10d vs 7 days?
I found this in the supplemental material from the study: “Our proposed margins of non-inferiority (NI) is in keeping with Sorbello’s suggested standard absolute NI margin of 7% with respect to all-cause mortality for clinical trials that assess antibiotic
drug efficacy in critically ill patients” (29). Thoughts?
Haven’t heard of Sorbello.. but that suggestion makes no sense at all. You are supposed to set NI margins based on clinically acceptable margins. No clinician in the world would accept a 7% increase in mortality. We probably wouldn’t accept a 1% increase in mortality just to decrease antibiotics by 24 hours. This is why NI is just not an appropriate approach here. Many researchers / methodologist have argued NI should never be used for mortality, and others have argued it is an outright unethical trial design. Not sure I would go that far, but it was clearly used inappropriately here.
Go into the NI issues at more length in this post: https://first10em.com/you-dont-understand-non-inferiority-trials-and-neither-do-i/