Science publication is broken – one way to fix it

There are two massive problems with science publication, and maybe only one way to fix them. Any long time reader of this blog will have encountered me saying something like “this study should never have been published.” (See, for example, Andexanet Alfa: More garbage science in the New England Journal of Medicine.) Although the statements are somewhat tongue in cheek, I think the underlying message is important: some data is so biased that it will predictably mislead clinicians and result in bad patient care. Publishing scientifically misleading articles seems like a really bad, perhaps even unethical practice.

However, every time I make a comment like that, someone correctly points out the second huge problem with science publication: unpublished data. “How can you suggest that studies shouldn’t be published!? Doesn’t that just create publication bias?” These comments are entirely correct. Unpublished data is a massive problem. It completely skews the medical literature, making ineffective or harmful medications seem useful by hiding most of the negative trials (think Tamiflu). 

In addition to biasing our entire understanding of medical science, unpublished data is also unethical. Trial participants take a big risk engaging in research. We give them an experimental chemical or intervention. We know that a proportion will experience an adverse event, but most of these trials don’t demonstrate any benefit. The only justification we can offer participants is knowledge – the hope that their sacrifice will improve the health of future patients. When trial data is hidden, that justification is lost, voiding the consent of research participants and rendering the entire research process unethical. (Hiding the results of trials behind a paywall is a grey area, but I am sure it would also upset a lot of trial participants, especially when the science was funded by their tax dollars.)

These two problems are contradictory. I don’t think any research should remain unpublished, but I honestly think a lot of the research that is currently being published is garbage and that we would be better off without it. The cognitive dissonance should cause my head to explode.

Some people might claim that the solution is simply critical appraisal. Let researchers publish all the crap science that they want, because clinicians can read the studies and ignore the bad ones. Unfortunately, this approach clearly doesn’t work. There are just too many examples of practices being widely adopted on the basis of bad science. We still have no idea if thrombolytics work for ischemic stroke, but it has become a “standard of care”. We are seeing formal recommendations to use idarucizumab and andexanet alfa, despite no evidence that either works and a real risk of harm. (Baugh 2019) Examples of bad science driving clinical care are nearly endless.

The proliferation of garbage science also makes it impossible for a practicing clinician to keep up. I spend far more time reading the literature than most other clinicians and I can barely scratch the surface. Good science is lost in a sea of trash. 

To make matters worse, once results are incorporated in meta-analyses, poor quality methods become far less evident. We are then presented with a single number as if it is “the truth”. (Even studies that have been formally retracted because of fraud continue to find their way into meta-analyses, significantly influencing the results.) (Garmendia 2019)

Patients are harmed by bad science. I don’t think the current practice of relying soley on critical appraisal is good enough.

In my mind, there is only one solution to these contradictory positions. The good news is that it is simple. If people care about science, we could adopt it tomorrow. We just need to change the time at which studies are accepted for publication.

Journals should accept studies for publication before any data is collected.

The quality of a study has nothing to do with the results section. Science and critical appraisal lives in the methods section. The only reason to wait for the results to be collected before accepting a study is that for profit journals like the New England Journal of Medicine want to publish exciting results that will sell multiple reprints. The current system is designed to maximize profits, not support good science. Studies shouldn’t be selected based on whether or not the results are exciting. They should be selected based on having high quality methods that reduce bias as much as possible.

In my system, researchers would submit their methods section before any data was collected. The journal editors would decide whether the study addressed an important question and used valid methods to provide us with an answer to that question. If the methods are accepted, the journal guarantees publication of the results, whatever they happen to be.

Negative results will be just as likely to be published as positive results. The researcher knows that their work will be published in a high quality journal once the data is collected. The ideal system would even allow researchers to receive feedback from the community about their methods before data is collected (instead of the current system where we all just fruitlessly critique the methods after the fact).

Because unpublished data is unethical, researchers couldn’t start a research project until a journal had agreed to publish the results. Shifting the acceptance for publication to the beginning of the process could eliminate a huge portion of the garbage science being produced. 

Of course, there are other issues that need to be solved simultaneously. We need to eliminate the money in science. The cynics out there are probably thinking that big drug companies will still be able to buy their way into the New England Journal no matter how crappy their methods, and they are probably right. For profit science journals should probably be eliminated altogether (a topic for a future rant). We need to eliminate predatory journals. My system doesn’t instantly solve these issues, but I do think it makes the solutions a little more attainable. (Research funding could be tied to publication in quality journals, which would be tied solely to the quality of the research methods.)

The apparatus of science publication is currently broken. It allows too much garbage science to be published. At the same time, it allows too much important science to remain unpublished. The fix is not going to be easy, but it is relatively simple. We just need to find the will to foster the necessary scientific revolution. 

References

Baugh CW, Levine M, Cornutt D, et al. Anticoagulant Reversal Strategies in the Emergency Department Setting: Recommendations of a Multidisciplinary Expert Panel Annals of Emergency Medicine. 2019;

Garmendia CA, Nassar Gorra L, Rodriguez AL, Trepka MJ, Veledar E, Madhivanan P. Evaluation of the Inclusion of Studies Identified by the FDA as Having Falsified Data in the Results of Meta-analyses: The Example of the Apixaban Trials. JAMA internal medicine. 2019; 179(4):582-584. [pubmed]

Photo by Patrick Fore on Unsplash