How is the BMJ different from MAD Magazine? Moving beyond peer review

How is the BMJ different from MAD Magazine? Well, I have never found a hidden picture by folding the pages of the BMJ, and although the editorials are occasionally pithy, they really can’t compare to the entertainment value of Spy vs Spy. On the other hand, the BMJ is considered a trustworthy collection of important medical information. Both are just words on paper written by people, so why is the BMJ considered more trustworthy? Peer review is supposed to be the answer. Nothing is published in the BMJ without first being reviewed for quality and truth by a panel of experts. But how well does this process of peer review work?

“Dismally” is probably the first word that comes to my mind. Peer review can’t possibly address the systemic problems we have fostered by allowing science to be done by those with a vested interest in the outcomes, but even for evaluating the papers at face value peer review seems to be a poor tool. We rely on content experts, but generally not methodology experts – and its the methodology that makes or breaks a paper. Richard Smith, a former editor of BMJ, had this to say about peer review: “It is slow, expensive, ineffective, a lottery, biased, incapable of detecting fraud and prone to abuse”. (Smith 2004)

Maybe theory and expert opinion don’t convince you (good), but how about some empirical evidence? Baxt and and colleagues (1998) rather mischievously sent out a purposefully flawed manuscript to reviewers. There were both major and minor flaws in the paper. How good were the reviewers? More than 2/3rds of the major errors were never identified by the reviewers. Not great.

Reviewers aren’t even consistent. Peters and Ceci (1982) took 12 papers that had already been published in major psychology journals and just resubmitted them to the same journals, but using different author names and institutional affiliations. Only 3 of the 12 were recognized as duplicates. Of the 9 that were not recognized as duplicates, 8 papers were rejected despite the fact the same paper had been accepted just 1-3 years previously.

There are a lot of things we need to fix about the way that medical research is currently being done. I will probably rant more about those in the future, but the purpose of today’s rant is to discuss the most important solution we have until we are willing to consider a complete overhaul of the entire medical research system: post-publication review.

Peer review is not perfect. It probably has its role, but there is no way that 2 or 3 reviewers can catch everything. Should peer reviewers be trained methodologists? For sure. Should they be required to perform simple tasks, like comparing the manuscript to the initial trial design on ClinicalTrials.gov? Absolutely. Can we expect them to be perfect? Of course not.

However, as soon as a trial is published, the number of reviewers jumps for 2 or 3 to a couple thousand. We can and should have our say about the quality of published research. That is one of the major advantages of the FOAM movement. Look at the brilliance of Ryan Radecki at EM Lit of Note, Rory Spiegel at EM Nerd, Anand Swaminathan (@EMSwami), Salim Rezaie (@srrezaie), and so many more.

But you don’t have to be one of those celebrities to have your say. If you want to take an active and important role in reviewing and critiquing the emergency medicine literature, it is easy to get involved. I would suggest checking out and participating in Ken Milne’s Hot Off the Press project at theSGEM. He takes a brand new publication, does the normal (excellent) SGEM critical appraisal, and then opens the floor to comments from the audience that can be directed directly at the author of the paper. All this while cutting the knowledge translation window down to less than 1 month. Pretty incredible.

The most recent SGEM HOP looked at this systematic review on femoral nerve blocks in the CJEM. Have a read and leave your comments for Ken and the lead author Brandon Ritcey over at the SGEM.

Get involved. Join the discussion. I think that the most important outcome of the FOAM revolution is bound to be the changes we will see in dissemination and analysis of medical literature through our worldwide, asynchronous #FOAMed journal club. Post publication peer review is incredible and is here to stay.

---

If you are interested in finding more EBM reviews, check out these resources:

Individual article critical appraisals

Large topic EBM deep dives

If you want some more EBM resources, this might help.

References

Smith Richard. Travelling but never arriving: reflections of a retiring editor BMJ 2004; 329 :242

Baxt WG, Waeckerle JF, Berlin JA, Callaham ML. Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Annals of emergency medicine. 32(3 Pt 1):310-7. 1998. PMID: 9737492

Peters DP, Ceci SJ. Peer-review practices of psychological journals: The fate of published articles, submitted again. Behav Brain Sci. 5(02):187-. 2010.

Ioannidis JP. Why most published research findings are false. PLoS medicine. 2(8):e124. 2005. PMID: 16060722

Millard WB. The Wisdom of Crowds, the Madness of Crowds: Rethinking Peer Review in the Web Era. Annals of Emergency Medicine. 57(1):A13-A20. 2011. [free full text]

Berger E. Peer review: A castle built on sand or the bedrock of scientific publishing?. Annals of Emergency Medicine. 47(2):157-159. 2006. [free full text]