Morgenstern, J. EPIC-HR: Some underwhelming data on Paxlovid, First10EM, April 12, 2022. Available at:
https://doi.org/10.51684/FIRS.127210
My initial evidence update for Paxlovid (nirmatrelvir/ritonavir) contained exactly no evidence, because nothing was published, despite recommendations at that time from multiple organizations. Sadly, two years into the pandemic, a large portion of the medical community still seems willing to act based solely on drug company press releases. The EPIC-HR trial has now been published, so we can at least discuss the science, which unfortunately is a bit of a mess. At the end of the day, I am no more certain that Paxlovid truly helps than I was before this trial was published.
The paper
Hammond J, Leister-Tebbe H, Gardner A, Abreu P, Bao W, Wisemandle W, Baniecki M, Hendrick VM, Damle B, Simón-Campos A, Pypstra R, Rusnak JM; EPIC-HR Investigators. Oral Nirmatrelvir for High-Risk, Nonhospitalized Adults with Covid-19. N Engl J Med. 2022 Feb 16:NEJMoa2118542. doi: 10.1056/NEJMoa2118542. Epub ahead of print. PMID: 35172054 NCT04960202
The Methods
This is a phase 2/3 double-blind, placebo controlled RCT.
Patients
Adult patients with confirmed, symptomatic COVID-19, within 5 days of symptoms onset, and at least one high risk feature for progression to severe disease.
Exclusions: COVID vaccination, prior COVID infection, anticipated need for hospitalization within 2 days, receipt of convalescent plasma.
Intervention
Paxlovid (nirmatrelvir plus ritonavir 300mg/100mg) orally every 12 hours for 5 days.
Comparison
Placebo.
Outcome
The primary outcome was a composite of all cause mortality and disease specific hospitalization at 28 days. They used a modified intention to treat analysis that only included patients who received the drug within 3 days of symptoms onset, despite their inclusion criteria.
The Results
They enrolled 2246 (of a planned 3000) patients. The median age was 46, 51% were male, 80% had a BMI over 25, 39% were smokers, and 33% had hypertension. 66% presented within 3 days of symptom onset.
For their composite primary outcome of hospitalization due to COVID-19 or death, there was a significant reduction with treatment (0.7% vs 6.5%). The results look similar in their secondary analysis looking at all patients (up to 5 days after symptom onset): 0.8% vs 6.3%.
They do not present any statistics around the overall mortality numbers. There were 0 deaths in the treatment group and 9 (1.3%) in the placebo group.
They don’t present overall hospitalization numbers at all.
There is not a difference in adverse events reported, but when adverse events are occurring in 25% of both groups, and can be difficult to pick out a smaller signal of harm from a novel, experimental agent.
My thoughts
Despite the fact that this is a double-blind RCT with promising results which have a lot of people excited, this trial has many problems that leave me very uncertain about the results.
This trial is completely run by Pfizer. The lead author, senior author, and many of the other authors are employees of Pfizer. Therefore, there is a massive conflict of interest that needs to be considered. As I discuss in my EBM bibliography, industry sponsored (let alone designed and run) studies are much more likely to find positive results for their own products. There is empirical evidence that these industry studies are biased or cheating. (Eg, Lexchin 2003) Therefore, we must always downgrade our confidence in studies with such obvious conflicts of interest. If, without the conflict of interest, you read this study and thought there was a 60% chance that Paxlovid works, the conflict of interest means that you probably need to drop that estimate to less than 40%, and perhaps much more. (Of course, those numbers are entirely made up.) Replication is always essential in medicine, but much more so when the person running the trial is also trying to sell you something.
It has been a recurrent problem in COVID research, but this trial uses the incredibly biased and completely irrelevant outcome of “COVID-19 related hospitalizations”. This transforms a relatively objective outcome into a completely subjective one. It means that patients can be admitted to hospital for severe drug reactions, and simply be ignored. And it means that the manuscript provides us with a meaningless number. It doesn’t tell us how many people were admitted to hospital, which is the outcome that doctors and patients actually care about. It only tells us the subset of admissions that they thought were due to COVID (which we all know is incredibly hard to distinguish). I find this especially problematic in this trial, because they don’t even present the all-cause hospitalization numbers. (They don’t even list the real hospitalization numbers in the supplement.) There is no way to know from this manuscript whether paxlovid decreased total hospitalizations, which is the only purported benefit. (For that matter, based on this publication, it is entirely possible that Paxlovid actually increases hospitalizations. Based on the numbers of “COVID related hospitalizations”, and the numbers of adverse events listed, I doubt this is an issue, but there is just no way to know.)
Compared to the completely irrelevant, subjective, and biased outcome of disease specific hospitalization, the 1.3% reduction in mortality would definitely be clinically important. I don’t understand why they don’t present statistics for this finding. (When I plug the numbers into an online calculator, I get a p value of 0.002 for this outcome.) In my mind, this number is really the only promising part of the published manuscript. The other sources of bias still limit my certainty, and it only applies to a very select group of patients who mostly don’t look like the patients I am seeing, but the potential mortality benefit is a finding worth pursuing in future research.
They used a modified intention to treat analysis that only included patients who received the drug within 3 days of symptoms onset, despite including patients up to 5 days after symptom onset. Their statistical plan is not described anywhere in their trial registry, and they don’t really explain why they limited their statistical analysis to patients within 3 days of symptoms despite including patients out to 5 days. I have never seen a study adjust their inclusion criteria within their statistical analysis like that, and in an industry run study, it makes me concerned about potential p-hacking.
The trial was stopped early, enrolling 2246 of the planned 3000 patients. (The number gets even smaller, because they only included 1379 patients in their modified intention to treat analysis.) They did stop at a planned interim analysis, but early termination means that the benefit is likely to be overestimated.
There are a number of secondary outcomes listed in the trial registry, such as duration of symptoms, severity of symptoms, hospital length of stay, and ICU length of stay, which are not listed in the manuscript, which is a form of publication bias. (Considering that Pfizer is the one writing up these results, you can bet that these results would have been listed if they had made Paxlovid look good.)
As I discuss in the longer philosophy interlude in the budesonide article, we must account for pretest probability when assessing new studies. During the pandemic, we have thrown so many different chemicals as COVID-19, we should expect a number of studies to be positive by chance alone. (The graph below from a recent publication in Nature gives a good sense of the low pretest probability required for any individual agent.). Therefore, a single positive study will never be completely convincing. When you add the fact that this is an industry run study with a primary outcome that is inherently problematic, I think that there is less than a 50% chance that these findings are replicated in independent high quality research.
Ultimately, I am just not sure what to do with this data. Outside of a pandemic, I think it is very clear that we would not recommend this drug at this time, and instead demand more evidence. An industry run trial with this many problems is simply not enough to safely base practice on. However, the pandemic may change the math, and the mortality finding might be important. (I honestly don’t know if the pandemic should change our decisions. It worries me to abandon our core EBM principles. If we are using different calculations during the pandemic, I think it is essential that we make that completely clear about it, because otherwise these practices will become forever entrenched, even when the pandemic is finally over.)
Considering the high probability that future independent research refutes these findings, we should be very careful in deciding who should receive this medication. If you wanted to argue that we shouldn’t be prescribing it at all, it probably wouldn’t take much to convince me that you are right. However, presuming that we are going to be prescribing it based on these uncertain but promising results, the significant uncertainty means we really should limit prescriptions to the population in whom it was studied. The data doesn’t seem strong enough to extrapolate to other, unstudied populations. Therefore, until further studies are published, I think the only patients who should be considered for paxlovid are unvaccinated adults with confirmed COVID-19, within 5 days of symptoms onset, and with at least one high risk criteria for progression to severe COVID-19.
There is a second RCT that has been reported through a press release. Based on their trial registry, the EPIC-SR trial looked at standard-risk patients, and focused on symptom resolution rather than hard outcomes. (Sounding like Tamiflu.) I will probably cover it when it is published, but based on the fact that it is essentially a mirror to this study, run by Pfizer, I expect many of the problems to be the same.
(I think we would do well to remember how much money was spent during H1N1 on the ‘miracle drug’ Tamiflu, and just how well that turned out.)
Bottom line
Considering the financial conflict of interest and multiple significant sources of bias, I think this is low quality evidence. The only benefit claimed is in a biased, subjective, and completely non-patient oriented outcome. Outside of a pandemic, I would definitely not prescribe Paxlovid. Whether you think this evidence is strong enough to warrant treatment will depend a lot on how much you think a pandemic changes the rules of evidence based medicine.
References
Hammond J, Leister-Tebbe H, Gardner A, Abreu P, Bao W, Wisemandle W, Baniecki M, Hendrick VM, Damle B, Simón-Campos A, Pypstra R, Rusnak JM; EPIC-HR Investigators. Oral Nirmatrelvir for High-Risk, Nonhospitalized Adults with Covid-19. N Engl J Med. 2022 Feb 16:NEJMoa2118542. doi: 10.1056/NEJMoa2118542. Epub ahead of print. PMID: 35172054
Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003 May 31;326(7400):1167-70. doi: 10.1136/bmj.326.7400.1167. PMID: 12775614
9 thoughts on “EPIC-HR: Some underwhelming data on Paxlovid”
Wonderful analysis as always. Will share this with my group treating a substantial number of COVID + patients with Paloxvid and it’s even more dubious buddy molnupiravir. Thanks for the incredible amount of time and effort you put into these analyses.
Very good Sir and very ….true.. I find subjective exclusion criterions a very likely source of bias. If we want to use paxlovid we’ll must repeat choice criterions used for patients in study but this is impossible. Furthermore…. in modified intention to threat, excluding patients, on a personal opinion, may introduce bias and lead to misleading results.
Always appreciate the thoughtful reviews. In this case, I’m not sure the strenuous complaint about omitting all cause hospitalizations is justified. Looking at the full 5 day initiation data set , the NEJM article reports primary outcome (essentially Covid hospitalizations) in 66/1046 PBO (6.4%), 8/1039 Paxlovid (0.78%) . In table 2 they report Serious Adverse Events (usually defined as any cause of hospitalization, life threatening event or fatality) with 18/1109 Paxlovid (1.6%), and 74/1115 Placebo (6.6%). Presuming SAE reported as they usually are, this would include Covid and non-Covid hospitalizations.
Different denominators is unfortunate, but setting that reporting choice aside, the math would point to maybe 10 extra hospitalizations in the Paxlovid arm, and 8 extra in the placebo arm, with vast majority of the SAEs (and therefore the hospitalizations) being the primary outcome events.
There are certainly many issues with this being only current study available for us (excluding Cyp3A4 meds, excluding vaxxed participants, etc), but if SAE is reported right, I don’t think omitting all cause hospitalizations is as big as you make out here.
Thanks for the comments
Although I agree that the numbers may work out fine here, the bigger concern is in the trial design. They designed their trial to be fundamentally flawed from the outset. They chose an primary outcome that they knew would favour their drug and downplay any harms. That is an important critical appraisal point when assessing the quality of a trial, whatever the results end up being.
Isn’t the coding of ‘serious adverse events’ a de factor all cause hospitalisation analysis?
I agree with all your concerns, and I am also frequently baffled by the low quality of publications, omitting such important details and tolerating such blatant sidesteps in methodology.
However, given the information that we have at hand, it seems much more likely that this drug is beneficial than harmful, so unless the drug is prohibitively costly, I would encourage my patients to take it.
Considering the primary results were for 3 days only, shouldn’t your (limited) recommendation be for 3 days of symptoms, not 5 days (were there enough patients in the 5 day group to power this recommendation?).
Thanks for the excellent article.
Spot on critical analysis. I add two more points: (1) Subjects were unvaccinated and without prior infection (2) The study ended December 10, 2021 when delta was dominant and the far different omicron was just beginning to take over. i.e. the study’s subjects were infected by the rather more virulent delta variant. Both of these points exaggerate the value of the intervention probably by an enormous decree. I’d only consider prescribing Paxlovid to an unvaccinated person who arrives by time machine from December 2021 or earlier. Otherwise, the medication has no demonstrated value for the present circumstances. Since Paxlovid gained EUA, I’ve never consider prescribing it: None of dozens of patients I’ve cared for reflect the study subjects infected by pre-omicron coronavirus. None are worse than a bothersome “head cold” and all have recovered within 2-3 days on “chicken soup” therapy.