Crappy guidelines and COVID-19

The pressure cooker of COVID-19 has brought out examples of both the best and the worst in medicine. We need to spend more time focusing on the best – the incredible creativity, innovation, teamwork, compassion, and dedication demonstrated throughout this crisis. However, we also need to consider the lessons that we can learn from COVID-19. For example, we have seen scientific principles abandoned during this crisis, as people rapidly adopt novel treatments with little to no evidence. Hopefully, as we reflect on these stressful times, the importance of evidence based medicine will become clear. I also hope that we will finally address a long standing problem in medicine: poorly written guidelines. 

Medicine has a long history of creating guidelines that make definitive statements without adequate discussion of science or logic that underlies those statements. Guidelines rarely including satisfactory discussions of the flaws in underlying studies, and almost never include discussion about controversy or dissenting opinions. They typically don’t include confidence intervals, absolute risk reductions, or discussion about potential harms. They rarely point out alternative choices. Most don’t have mechanisms for feedback from the community. Potential conflicts of interest are seldom mentioned and guideline panels generally represent a very narrow set of perspectives. All too often, they don’t even reference the underlying science. In short, as I have discussed before, most of our guidelines suck.*

In normal times, these problems are an annoyance. Guidelines aren’t laws. They are meant to guide, not decree. We can read the science for ourselves, incorporate the perspectives of our patients, and deviate from the guidelines when necessary.

However, the COVID-19 crisis has highlighted the disastrous consequences of poorly constructed guidelines. Bad guidelines have increased stress tremendously. They have created conflict. Rapidly changing guidelines, or conflicting guidelines from different authorities, have increased confusion and resulted in inconsistent care. I really hope that we can learn from our mistakes. 

The last few months have seen rapid policy changes at every hospital in the world. New guidelines seem to come out almost every day, and the new guidelines frequently contradict older ones. Although many find this confusing, I actually think it is a good thing. We need daily guidance when faced with a new and rapidly emerging pandemic. We need to be willing to rapidly change our practices. We should always be willing to change our minds when new evidence becomes available. Being wrong is not a problem. It only becomes a problem if we are unwilling to admit our errors or too stubborn to change our ways. 

One special advantage of the skeptical attitude of mind is that a man is never vexed to find that after all he has been in the wrong.
– William Osler

Unfortunately, the poor construction of guidelines makes it harder to change clinical practice, and increases confusion and stress when changes are made. This happens for a few reasons. Often the initial guidelines sound definitive and do not recognize the underlying scientific uncertainty or inform readers that changes are possible (or maybe even likely). Too often, guidelines don’t explain their reasoning. They don’t discuss the science or logic that led the authors to make the recommendations that they do. Sometimes, the guidelines don’t even contain references. They are black boxes. Trust is expected, but there is no mechanism for verification. This is particularly problematic when multiple guidelines are released simultaneously with directly conflicting recommendations. (Expert opinion is great, but quickly loses all authority in the face of a second expert with a different opinion.)

One example from the COVID-19 pandemic is the conflicting guidelines about chest compressions. The government of Ontario released a guideline entitled “Personal Protective Equipment (PPE) use during the COVID-19 Pandemic”. Although cardiopulmonary resuscitation is listed as an aerosol generating medical procedure, they make the specific note that “chest compressions and cardioversion/defibrillation are not considered AGMP”. Unfortunately, there is no citation or explanation for this statement.** This statement is in direct conflict with guidelines from other major organizations. For example, the ILCOR statement is, “we suggest that chest compressions and cardiopulmonary resuscitation have the potential to generate aerosols (weak recommendation, very low certainty evidence).” Similarly, the AHA states that “the administration of CPR involves performing numerous aerosol-generating procedures, including chest compressions, positive pressure ventilation, and establishment of an advanced airway.” Their first recommendation is “before entering the scene, all rescuers should don PPE to guard against contact with both airborne and droplet particles. Consult individual health or emergency medical services (EMS) system standards as PPE recommendations may vary considerably on the basis of current epidemiologic data and availability.” The AHA and ILCOR guidelines are based on a literature search, and contain a brief discussion of their rationale, but can’t reference evidence for their position (because there really isn’t any), and still probably don’t adequately address the uncertainty of their recommendations. The result is a conflict that leaves clinicians confused, generates anxiety, and may create conflicts in the workplace.

Neither of these groups is necessarily wrong. Nobody has made a mistake. The evidence is so weak that it can realistically be interpreted in many ways. The authors of these documents are all incredibly intelligent and should be commended for their effort. You can’t blame the guideline authors for the fact that evidence doesn’t exist.

However, I think we should find fault with the defective format of these documents. I don’t blame the authors, because most guidelines follow a fairly traditional format, but there is a major problem. Without citations, there is no mechanism to resolve disputes that arise as guidelines are implemented clinically. There is no mechanism to check the authors’ work. (We are all human, after all, and everyone makes mistakes). Without explanations, there is no way to know if guidelines truly apply to the unique patient in front of you. Without explanations, there is no way to determine who is right when guidelines conflict. Most importantly, despite being the most honest answer, our guidelines almost never say “we don’t know”.

I think the authors of all the above guidelines would probably freely admit that they don’t know. The AHA and ILCOR documents acknowledge that the level of evidence is weak. However, it is easy to miss or ignore the level of evidence found in brackets at the end of a statement, especially when that statement sounds pretty definitive. Nothing is nearly as powerful (or honest) as the simple statement, “we don’t know.” 

If the evidence does not support straightforward conclusions, pretending it does is worse than admitting uncertainty. (Lenzer 2013)

Without a clear acknowledgement of uncertainty, these documents are frequently translated into relatively authoritarian dictates. As these documents are simplified for education and implementation, the “(weak recommendation, very low certainty evidence)” is often left out. Protocols are created without any acknowledgement of the uncertainty. Practicing clinicians are left to believe that the recommendations are black and white.

The problem is not isolated to chest compressions. Almost every guideline I have seen during COVID-19 has suffered from these same flaws. Infection control practices, screening protocols, and treatment guidelines have all offered definitive advice without necessarily acknowledging the fundamental uncertainty, discussing their rationale, or referencing the underlying science. 

The result is anxiety, confusion, and conflict. Practices vary country to country, hospital to hospital, and potentially even ward to ward. Authoritarian guidelines offer no recourse for discussion when clinicians have questions or concerns. The guidelines tell us what to do, but refuse to comment on why.

Unfortunately, when we don’t acknowledge uncertainty in our guidelines we generate unnecessary stress when processes need to change. One policy is replaced overnight with a new, completely contradictory policy. Staff don’t know why the changes were made. They get anxious and they lose trust.

If we had acknowledged with the first policy that there is a great deal of uncertainty, and that we will update our policies as soon as more information becomes available, staff would be expecting changes. They would likely encourage changes, because they know that updates are a sign that someone is constantly looking out for their best interests. If the new guidelines had clear explanations (preferably with citations) of why changes were made, anxiety would decrease. But when the changes are made without warning and without explanation, they seem capricious rather than well reasoned. Trust is eroded and anxiety is increased. 

This is not a COVID problem. Medical guidelines have been broken for a long time. However, I am hopeful that as we learn our lessons from this COVID pandemic, we can finally get around to fixing them. 

Some Criteria for Good Guidelines

• A clear distinction between recommendations with strong evidence and those without.
• A discussion of harms alongside benefits.
• The use of absolute numbers or NNTs and the inclusion of confidence intervals.
• Clear references to the underlying science.
• Clear statements of uncertainty when it is present.
• A mechanism for feedback from all important stakeholders.
• The ability to update rapidly when new evidence arises.
• Inclusion of any dissenting opinions, if they exist.
• When a topic is known to be controversial, guidelines should actively seek involvement from authors on every side of the debate.
• The exclusion of authors with conflicts of interest and a clear discussion of how the panel of authors was selected. Authors should include both content experts and science or methodology experts, with the science experts taking the lead.
• If possible, reference to and discussion of any conflicting guidelines.

Please feel free to to add more suggestions for high quality guidelines in the comments below.

References

Couper K, Taylor-Phillips S, Grove A et al. COVID-19 infection risk to rescuers from patients in cardiac arrest. Consensus on Science with Treatment Recommendations [Internet] Brussels, Belgium: International Liaison Committee on Resuscitation (ILCOR), 2020 March 30. Available from: http://ilcor.org

Edelson DP, Sasson C, Chan PS, et al. Interim Guidance for Basic and Advanced Life Support in Adults, Children, and Neonates With Suspected or Confirmed COVID-19:From the Emergency Cardiovascular Care Committee and Get With the Guidelines -Resuscitation Adult and Pediatric Task Forces of the American Heart Association in Collaboration with the American Academy of Pediatrics, American Association for Respiratory Care, American College of Emergency Physicians, The Society of Critical Care Anesthesiologists, and American Society of Anesthesiologists: Supporting Organizations: American Association of Critical Care Nurses and National EMS Physicians Circulation. 2020;

Lenzer J, Hoffman JR, Furberg CD, Ioannidis JP,  . Ensuring the integrity of clinical practice guidelines: a tool for protecting patients. BMJ (Clinical research ed.). 2013; 347:f5535. [pubmed]

Ontario Helath. Personal Protective Equipment (PPE) use during the COVID-19
Pandemic. Recommendations on the use and conservation of PPE from Ontario Health. 2020. Available online here.

Footnotes

*There are, of course, some exceptions. These generalizations are bound to overlook some specific examples of incredibly well done guidelines. However, on average, I think these generalizations are consistent with the majority of guidelines I encounter.

**Again, I think it is important to emphasize that I am not criticizing the authors of these specific guidelines. Rather, I am criticizing the way that almost all guidelines in medicine are written. I don’t know most of the authors here, but Dr Howard Ovens is one of the most respected emergency doctors in Canada. He is brilliant, caring, and well meaning. This essay is not about Dr. Ovens being right or wrong. It is about making the science and logic behind the guideline he helped create transparent and more usable for the practicing clinician.

Some added context from Dr. Ovens:

Justin, overall great rant! And you made some very generous comments about me, which are appreciated but not necessary – you can question me or my work or anyone’s – without qualification as long as it is not an ad hominem attack which you never stoop to. With respect to the reference you cite, we do in fact cite our source (14Release date: March30, 2020Appendix A: Aerosol-Generating Medical Procedures, adapted from the Toronto Region Hospital Operations Committee IPAC Consensus List of Aerosol-Generating Medical Procedures (AGMP) if not primary references. We did that because we did not think we were constituted, nor had the time or expertise to be a scientific panel creating “guidelines”. We provided recommendations (probably a distinction without a difference there) and wanted to ensure our recommendations were consistent with other major Ontario documents mainly to avoid the confusion you so articulately decry. We also did clearly say that the info in our document was subject to change in a rapidly changing environment. I say the above not to defend our work, it can stand or fall on its own merits, but to give the citation a bit of a broader context for your readers to judge. Keep up the great work and best regards, Howard