Remdesivir: The ACTT-1 Trial

The first RCT of remdesivir for COVID-19 was not encouraging, but even as it was being discussed, we knew that another trial existed, and we were told that the results were positive. The results of that trial (ACTT-1) are now out. 

The paper

The ACTT-1 Trial: Beigel JH, Tomashek KM, Dodd LE, et al. Remdesivir for the Treatment of Covid-19 — Preliminary Report. N Engl J Med. 2020; DOI: 10.1056/NEJMoa2007764 NCT04280705

The Methods

ACTT-1 is a multi-center, double-blind, placebo controlled trial run at 60 trial sites across 10 countries.

Patients

Hospitalized, adult patients, with laboratory-confirmed COVID-19. 

Intervention

Remdesivir 200 mg IV on day 1 and then 100 mg IV for 9 days, or until hospital discharge or death, whichever came first.

Comparison

Placebo. (Due to shortages, the European sites were unable to procure a matching placebo, and used normal saline in an opaque bag instead. It is unclear whether this resulted in any unblinding.)

Outcome

The primary outcome was changed after the trial started, but before analyzing the data. The final primary outcome was the time to recovery, defined as the first day during the 28 days after enrollment that the patient fell into the lowest 3 categories on their 8 category ordinal scale:

1. Not hospitalized, no limitations of activities
2. Not hospitalized, limitation of activities, home oxygen, or both
3. Hospitalized, not requiring oxygen or medical care (ie, kept in hospital only for infection control purposes)
4. Hospitalized, not requiring oxygen, but requiring ongoing medical care
5. Hospitalized and requiring supplemental oxygen
6. Hospitalized and requiring noninvasive ventilation or high-flow oxygen devices
7. Hospitalized and receiving invasive mechanical ventilation or ECMO
8. Dead

The Results

They enrolled 1063 patients overall. About 10% of both groups withdrew from the trial because of serious adverse events. They stopped data collection early, so the presented results only represent 731 of those 1063 patients. The patients were a mean age of 59 and 64% male. Most came from North American (80%). Most had one (27%) or more (52%) comorbidities. The median number of days from symptom onset to randomization was 9. 

Final primary outcome: Patients in the remdesivir group had a shorter time to recovery than those in the placebo group (median 11 days vs 15 days, p<0.001, rate ratio 1.32, 95% CI 1.12-1.55).

Original primary outcome: The odds of improvement at 15 days were also better with remdesivir (OR 1.5, 95% CI 1.18-1.91, p=0.001).

No benefit was seen in critically ill patients. (I have previously discussed the difficulty of timing for antiviral drugs.)

Mortality was not statistically lower, but the point estimate is on the side of improvement with remdesivir. Mortality by 14 days was 7.1% with remdesivir and 11.9% with placebo. The hazard ratio was 0.7 (95% CI 0.47 – 1.04). 

My thoughts

ACTT-1 is a great trial. To get a double-blind, randomized trial done in just a few months in the middle of a pandemic is a tremendous feat, and these authors deserve an incredible amount of credit. These results are very promising, but the problem in trying to interpret a single trial is there is no way to be sure if the results we are seeing represent reality, bias, or a chance finding.

There are a number of potential sources for bias in this trial. The change in primary outcome has received a lot of attention already (before the trial was even published). The original primary outcome just looked at everyone’s status on day 15, but as COVID progressed it became obvious that many people were having much longer courses than that. They changed the outcome early in the trial, without looking at the data to date. This is the best possible way to change a primary outcome and their logic makes a lot of sense. However, anytime researcher degrees of freedom increase (they are allowed to change things from the original protocol), you also increase the potential for bias and the probability of finding positive outcomes by chance alone. That being said, the original primary outcome was also positive in this trial. I am not overly bothered by this change.

The weirdest thing about this trial was the manner in which it was stopped. They had enrolled all of the patients required, but they hadn’t followed them for their entire course yet. So this trial was stopped early, with full outcomes only available for 69% of the patients. This could matter a lot, considering what they are measuring. For example, if remdesivir simply delayed bad outcomes like death rather than preventing them, those deaths wouldn’t be counted, because we are only counting early outcomes. This early stopping created an imbalance between the two groups in terms of patients with incomplete data (169 in the placebo group vs 132 in the remdesivir group). This could be because people in the remdesivir group were improving faster, but also could represent an important source of bias in the results. 

I think it was a mistake to stop this trial early, just like it was a mistake to release the results by press release before publication. I understand the enthusiasm for treatment options in the face of this pandemic, but it would have only taken a few weeks to complete the trial properly, and the results would have been far more sound. As it stands, all the remaining patients were allowed to become unblinded, and placebo patients can be transitioned to remdesivir, so the final data from this trial will not be all that helpful. Had the trial run to completion, we might have seen a definitive mortality benefit. Stopping the trial early ultimately necessitates more research, requiring more patients to be randomized if we want to be sure about the harm/benefit ratio of remdesivir. That seems like a huge cost for the sake of rushing this data to print a few weeks early. 

Publication bias is one of the biggest problems in modern medicine. We really need to ensure that every planned outcome is registered, and every registered outcome is published. When outcomes are hidden, the science is drastically biased. ACTT-1 has 27 secondary outcomes registered on clinicaltrials.gov, but the vast majority are not mentioned anywhere in the manuscript. That is problematic.

External validity may also be an issue. Although it is a multinational trial, because of the way the pandemic progressed, the majority of patients are from American sites. We need to be cognizant that the majority of these patients were enrolled in the hardest hit hospitals in the world, which could impact the baseline standard of care through staffing and supply issues. Furthermore, the American healthcare system is unique in the world, and outcomes seen there may not always extrapolate to other countries. 

The primary outcome is somewhat subjective. Although the categories seem objectively discrete, there is actually a lot of subjectivity that goes into every medical decision, such as whether someone needs oxygen or whether someone is ready to go home. A change from one level to the other on this scale, although it sounds good, may not represent a truly important outcome. 

The mortality outcome will probably get the closest attention in this trial. Although the result is “statistically insignificant”, I think the results should increase our belief that remdesivir could provide a mortality benefit. That isn’t to say that remdesivir does improve mortality, but if you thought there was a 5% chance of it working before this trial, the hazard ratio of 0.7 with a 95% confidence interval that is mostly on the side of benefit might cause you to revise that estimate up to 10%. Of course, statistical questions are always secondary to questions of bias. (There is no point focusing on precise statistical measures if the trial was unfair in the first place.) The biggest source of bias in this trial is the fact that they stopped collecting data early. This resulted in the loss of data in more patients from the placebo group. This would be particularly important if remdesivir is changing the time of death, rather than actually changing your chance of dying.

So how likely is it that the results seen here represent a true benefit from remdesivir? Given the sheer number of COVID-19 trials being run, it was inevitable that some would be positive. The issue is picking out those which are positive by chance alone, or through bias in their trial design, from those that are positive because they represent a true difference. It is impossible to make that distinction from a single manuscript. This is why replication is key in science. We will only know if this was a true difference if we see it replicated in future studies.

However, we can make an estimate as to how likely those future studies are to be positive (which might be important when trying to determine what to do clinically.) In order to do so, we need to know how likely we thought it was that remdesivir worked before ACTT-1 – our pretest probability. Personally, I think our pretest probability should be incredibly low. In the history of medicine, there has never been a treatment that improves patient oriented outcomes from a viral pneumonia. Nothing works. Furthermore, we already had one RCT of remdesivir, and it was negative (although that trial was under-powered, and the results actually look pretty consistent with this study). (Wang 2020) I would have set my pretest probability of remdesivir providing a patient important outcome at 1% (at most). With a p-value of 0.001, this trial would shift that 1% pretest probability to a 35% posttest probability that remdesivir works. (This calculation is just based on statistics, and completely ignores any bias that could make this result less reliable). Therefore, I am obviously unconvinced that remdesivir works.

Let’s say that you were more optimistic than I was, and thought that there was a 10% chance that remdesivir worked. (I don’t think you could possibly go much higher than 10%, given the previous negative trial and the fact that nothing has ever worked for viral pneumonia before). In that case, the p value of 0.001 translates into an 85% chance that remdesivir works. That is promising, but not definitive (especially since these calculations completely ignore the role of bias).

Whatever pretest probability you choose, it is pretty clear that this single trial doesn’t prove that remdesivir helps (especially when you account for bias). However, it is also clear that this trial should significantly shift our position towards the possibility of remdesivir working. Whatever your prior position, it is clear that further research is needed. Thankfully, that research is already underway. (It would be an absolute travesty and completely misunderstanding of science if those trials were stopped early based on these results). The only real debate is what we do clinically while waiting for more research. If you were an optimist before this trial was published, the results are clearly enough to prescribe remdesivir while waiting. Even if you held a more pessimistic position, like me, a 35% chance of benefit might be enough to convince you to try remdesivir. Of course, trials generally underestimate the harms from our drugs, and 10% of these patients discontinued the study drug because of significant adverse events, so we are a long way from being sure about the harm benefit ratio here.

Either way, this is not a miracle cure. There may be a small benefit, and even a small benefit could be huge in the face of a pandemic, but we need to be cautious in our interpretation of any single study.

Bottom line

This double-blind, placebo controlled trial of remdesivir for hospitalized adult patients with COVID-19 showed an improvement in time to recovery. These are promising results. They need to be replicated before we can be sure they are true, but they may warrant the use of remdesivir in some patients while waiting for further research.

Other ACTT-1 FOAMed

You can read more about pretest probability and Bayesian analysis here: EBM 2.0

Antivirals for COVID19

REBEL EM: Remdesivir ACTT-1: Adaptive COVID-19 Treatment Trial Part 1

The Bottom line: ACTT1

EM Nerd-The Case of the Partial Cohort

PulmCrit – Preliminary report on NIAID trial of remdesivir (ACTT-1)

References

Beigel JH, Tomashek KM, Dodd LE, et al. Remdesivir for the Treatment of Covid-19 — Preliminary Report. N Engl J Med. 2020 (ACTT-1)

Wang Y, Zhang D, Du G, et al. Remdesivir in adults with severe COVID-19: a randomised, double-blind, placebo-controlled, multicentre trial. The Lancet. 2020;