I hate guidelines (but they can improve and you can help)

A rant about guidelines (and an opportunity to help shape the future of the ILCOR guideline process)

I hate guidelines. I shouldn’t. In theory, summaries of the medical literature that are accessible to practicing clinicians could only be good. Unfortunately, in current practice, medical guidelines are too often biased, unscientific, overreaching, or misleading.

I started residency as a slave to guidelines. I read them all. I strictly followed every recommendation. But it didn’t take me long to become disillusioned.

My first insight into the fallibility of guidelines was on the topic of pediatric urinary tract infections. I started my family medicine residency rigorously following the current guidelines, meaning that a lot of febrile children were catheterized, and subsequently sent for painful medical procedures like the VCUG. (Roberts 1999) However, when I actually read the evidence for myself, I realized that almost everything I had been doing was wrong. I had been hurting children.

This pattern repeated itself over and over again. I would practice according to the newest guidelines, but whenever I took the time to read the evidence, I would be shocked to find that I wasn’t providing my patients with the best care. I was hurting patients; guidelines were hurting my patients.

Problems with guidelines

There are many problems with medical guidelines as they are currently produced and used.

Perhaps the biggest problem is that they are treated as gospel. People often act as if guidelines set the standard of care. Funding decisions and clinical bench marks are based on guidelines, often with little consideration of the underlying quality of evidence. (Lenzer 2013) Physicians often feel compelled to follow guidelines, even when they disagree with the recommendations. (Lenzer 2006; Woolf 1999) Unfortunately, because guidelines try to provide a single answer from complex and uncertain data, and different guidelines committees have different biases and conflicts, different guidelines on the same topic can often provide significantly different recommendations, provoking confusion and anxiety among clinicians and patients alike. (Feder 1994; Burgers 2002; McAlister 2007)

Scientific evidence is always complex and nuanced. There is almost never a single, easy answer. Evidence based medicine requires that we use clinical judgement and consider patient values. All too often, this complexity is lost in guidelines.

If the evidence does not support straightforward conclusions, pretending it does is worse than admitting uncertainty” (Lenzer 2013)

Guidelines rarely highlight scientific uncertainty. I have only seen confidence intervals presented in a clinical guideline one time. It is essentially unheard of for guidelines to discuss reasonable alternative options to their “final recommendation”. Although it is unlikely that every panel member perfectly agreed on every recommendation, discussions and debates are never included, so guidelines come across as infallible, unquestionable, or definitive. (Sniderman 2009)

Furthermore, too many guidelines are biased from the outset. Financial conflicts of interest are still rampant in medicine, including in guideline development. (Brindsley 2013; Choudhry 2002; Gale 2011; Holloway 2008; Kung 2012) Although the Institute of Medicine has recommended that guidelines eliminate (or reduce) conflicts of interest, one survey indicated that 70% of committee chairs and 90% of committee co-chairs have financial conflicts of interest. (Kung 2012) We know that financial conflicts, even if small, will significantly sway judgement. There is evidence that conflicted guidelines lead to drastically different recommendations. (George 2014) At their worst, some guidelines are little more than marketing tools for industry.

Guidelines also tend to severely limit the diversity of opinions expressed. Committees are comprised of a relatively small number of academics with similar backgrounds. (Woolf 1999) Community physicians are rarely represented. The unique needs and insights of rural communities are not included. The difficulties of low and middle income countries are frequently ignored. Guidelines present recommendations as if medical decisions are solely the purview of physicians, and generally overlook the insights of patients. (Lenzer 2013) These issues are made worse when the already small guideline committees are filled with individuals who all started with similar opinions – known as “panel stacking”. (Lenzer 2013; Sniderman 2009)

Solutions

Although there are many problems with our current guidelines, I still like the idea of easy to read, accessible summaries of the evidence for practicing clinicians. It is simply not feasible for clinicians to read every relevant study – not even me. We need summaries to have any hope of practicing evidence based medicine, but we need to drastically change how those summaries are produced and presented.

I think the most important step is to divide current guidelines into two completely separate documents. Currently, guidelines mix a few recommendations supported by strong evidence with a large amount of expert opinion. There is value in both, but they shouldn’t be in the same document.

In any current guideline, there are only a handful of very strong recommendations, based on multiple, consistent RCT level evidence. These are recommendations, like aspirin for STEMI, that we think should be followed almost all of the time. These very strong recommendations should be pulled out into a separate document, which would probably never be longer than 1 page. These would be the minority of medical practices with a high degree of certainty, and would be reasonable targets for attempts to limit (but not eliminate) variability in practice.

There is still a role for expert opinion and low quality, mixed, or biased evidence. However, mixing it into guidelines is often misleading or confusing, resulting in strong recommendations or funding decisions being based on low quality evidence. In areas that we don’t have strong evidence, I am fine with expert opinion, as long as it is clearly labelled as such at the outset, and it is clear that such recommendations in no way represent a standard of care.

In either case, it is important that guidelines not present their recommendations as “rules” that must be followed. Limitations of the science must be emphasized. Patient variations and values need to be discussed. It should be clear that it is appropriate for clinicians to deviate from guidelines based on their clinical judgment and depending on patient values.

Next, guidelines must embrace and discuss the uncertainty of science. Far too often, complex questions are simplified and presented as if there is a single strong answer. Weaknesses in the underlying evidence must be highlighted. All recommendations should be presented not just with absolute risk reductions or NNTs, but also confidence intervals. Potential harms must be discussed right next to potential benefits. Dissenting opinions from the guidelines committee, if they exist, must be published.

Financial bias is a huge problem in all of science, but is clearly problematic in guidelines. I believe that individuals with financial conflicts of interest should be completely banned from guideline committees. (This is a rather controversial statement. Others would argue for disclosing and managing conflicts of interest, but I think there is good evidence that disclosure can backfire, (Cain 2003) and maintain the firm position that individuals producing guidelines, especially considering the power guidelines currently have in medicine, should have no vested interest in the recommendations.)

The selection of guideline committees also needs to be addressed. Guidelines should be primarily about science, and therefore should be predominated by science experts (methodologists) rather than content experts. (Content experts are still necessary, but should be the minority). Guideline selection also needs to be more representative, including community practitioners, clinicians from rural settings, and representatives of low and middle income countries (as well as many other minority voices I am leaving off the list). Involving patients also makes a lot of sense.

Although a lengthy list, these are not difficult changes to make. Unbiased and nuanced summaries of clinical evidence are already being produced (just look at the FOAMed community). If the general medical community demands more from guidelines, I am optimistic that they will improve.

The ILCOR vision

And now is the time for us to demand more. Although there are many problems with our current medical guidelines, they are not a lost cause. But we all need to get involved if we want guidelines to be helpful tools that improve patient care.

Next month, I am going to be attending the annual ILCOR meeting. As far as guidelines go, I think the ILCOR recommendations (that get translated into guidelines by multiple agencies, such as the AHA and ERC) are pretty good. I always read them. But I was thrilled to hear that they are constantly trying to improve their process.

Drs Robert Neumar and Gavin Perkins, the current co-chairs of ILCOR, wants to change the way that the general medical community interacts with ILCOR and their process of generating consensus on science and treatment recommendations (CoSTRs) that are the foundation for international guidelines on cardiac arrest resuscitation and first aid. After a long conversation with Bob, I am pretty excited about the vision. (Aside from being invited to speak at their meeting, I have no ties to ILCOR. I am paraphrasing Dr. Neumar.)

At the core of the ILCOR philosophy are 2 ideas:

  1. It is better to draw on the expertise and wisdom of the entire medical community than just a limited number of hand-picked experts.
  2. Guidelines should not be created behind closed doors. The process should be open and transparent.

The current task forces that generate ILCOR CoSTRs are made up of 17 content experts with diverse methodologic, scientific and clinical expertise from across the world. Although this represents an incredible wealth of knowledge and expertise, a limited number of people can only provide a limited number of viewpoints. ILCOR wants to expand the range of voices that shape their treatment recommendations. Doctors working in the community often have important insights that differ from those in academic centers. Rural communities need a say on the impact and practicality of these guidelines in their setting. The same goes for low and middle income countries. Not every voice can be present at the meeting, but every voice can be heard.

ILCOR wants input at all stages of guideline development. Before the process even begins, the community could help decide what topics are most important to examine. During the systematic review, the community might help identify or critique important research. Once the guideline is written in draft form, the community should be able to comment and criticize. And once the guideline is finalized, the community can be involved in dissemination, discussing how it should be applied in practice, and pointing out when it needs updating.

ILCOR also wants to make this entire process more transparent.

These goals seem to align very well with the general ethos of FOAMed and the medical social media community.

Your opportunity

In a few weeks time, I am going to speak at the ILCOR meeting on the ways that they could use social media, or potentially partner with existing FOAMed resources, to improve their guideline process. Now is your chance to shape this process. Leave comments, and I will bring your ideas to the ILCOR committee.

  • ILCOR wants community feedback on its guidelines. How would you use social media or FOAMed to encourage that feedback?
  • How would you improve the guideline creation process, especially considering the desire to be transparent and interactive?
  • Do you have a FOAMed site? Any thoughts on how partnerships with FOAMed would improve the quality and usability of guidelines?
  • The most recent ILCOR CoSTRS (both final and draft) are on the ILCOR website. How would you improve this website to encourage feedback and discussion?

References

Bindslev JB, Schroll J, Gøtzsche PC, Lundh A. Underreporting of conflicts of interest in clinical practice guidelines: cross sectional study. BMC medical ethics. 2013; 14:19. [pubmed]

Burgers JS, Bailey JV, Klazinga NS, et al. Inside guidelines: comparative analysis of recommendations and evidence in diabetes guidelines from 13 countries. Diabetes care. 2002; 25(11):1933-9. [pubmed]

Cain DM, Loewenstein GF, Moore DA. The Dirt on Coming Clean: Perverse Effects of Disclosing Conflicts of Interest. SSRN Journal. 2003.

Choudhry NK, Stelfox HT, Detsky AS. Relationships between authors of clinical practice guidelines and the pharmaceutical industry. JAMA. 2002; 287(5):612-7. [pubmed]

Feder G. Management of mild hypertension: Which guidelenes to follow? BMJ : British Medical Journal. 1994;308(6926):470-471.

Gale EA. Conflicts of interest in guideline panel members. BMJ (Clinical research ed.). 2011; 343:d5728. [pubmed]

George JN, Vesely SK, Woolf SH. Conflicts of interest and clinical recommendations: comparison of two concurrent clinical practice guidelines for primary immune thrombocytopenia developed by different methods. American journal of medical quality : the official journal of the American College of Medical Quality. 2014; 29(1):53-60. [pubmed]

Holloway RG, Mooney CJ, Getchius TS, Edlund WS, Miyasaki JO. Invited Article: Conflicts of interest for authors of American Academy of Neurology clinical practice guidelines. Neurology. 2008; 71(1):57-63. [pubmed]

Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet institute of medicine standards: Two more decades of little, if any, progress. Archives of internal medicine. 2012; 172(21):1628-33. [pubmed]

Lenzer J. NIH secrets: study break. New Republic 2006 Oct 6. www.newrepublic.com/

article/national-institutes-health-fred-geisler-spinal-cord.

Lenzer J, Hoffman JR, Furberg CD, Ioannidis JP,  . Ensuring the integrity of clinical practice guidelines: a tool for protecting patients. BMJ (Clinical research ed.). 2013; 347:f5535. [pubmed]

McAlister FA, van Diepen S, Padwal RS, Johnson JA, Majumdar SR. How evidence-based are the recommendations in evidence-based guidelines? PLoS medicine. 2007; 4(8):e250. [pubmed]

Roberts KB. A Synopsis of the American Academy of Pediatrics’ Practice Parameter on the Diagnosis, Treatment, and Evaluation of the Initial Urinary Tract Infection in Febrile Infants and Young Children Pediatrics in Review. 1999; 20(10):344-347.

Sniderman AD, Furberg CD. Why guideline-making requires reform. JAMA. 2009; 301(4):429-31. [pubmed]

Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ (Clinical research ed.). 1999; 318(7182):527-30. [pubmed]

 

Cite this article as: Justin Morgenstern, "I hate guidelines (but they can improve and you can help)", First10EM blog, October 15, 2018. Available at: https://first10em.com/guidelines/.

Author: Justin Morgenstern

Emergency doctor working in the community. FOAM enthusiast. Evidence based medicine junkie. “One special advantage of the skeptical attitude of mind is that a man is never vexed to find that after all he has been in the wrong.” - William Osler

12 thoughts on “I hate guidelines (but they can improve and you can help)”

  1. Hi mate
    This is an awesome opportunity for the FOAM community. I recall sitting in the audience of the first SMACC listening to Simon Carley discuss “Anarchy in the UK” and the community has matured greatly since then. We do remain “reactionary” rather than proactive in matters like guidelines.. eg. The recent sepsis bundles and petitions

    The FOAM community has its strengths:
    – We are good at responding to new guidelines, picking them apart and bringing the evidence to light
    – generally FOAM is unbiased and transparent
    – there is diversity of thought (LMIC, rural, public / private)

    I would propose that the best way to involve FOAMed people would be to produce a draft guideline, with full references and engage the community. A kind of mega pre-publication review.
    The FOAM ecosystem will be quick to identify the weaknesses and the reality of practice that the guidelines often gloss over.

    Of course, somebody has to actually read and listen to all the FOAM – however it would be simple to submit a URL /link to the ILCOR or other website for consideration.

    What is needed is a simple “front door” for guidelines committees to engage the community in a proactive manner. I imagine that to a traditional committee that Twitter is not really a palatable option. However, you have shown that real human interactions can allow a conversation to begin. Well done
    C

  2. I enjoyed your rant Justin. Guidelines are something I have struggled with over my career too.

    Here are a few (five) problems:

    1) Misused to dictate care rather than guide our care
    While most guidelines have a disclaimer about not dictating care or being the standard of care they are often used this way. This can cause medical legal problems and create bad protocols. Administrators then use metrics based on weak evidence to pressure physicians to follow guidelines (often blindly). Clinicians are rewarded and punished based on concordance with guidelines.

    2) Conflicts of interest (financial and intellectual)
    A study in 2005 showed more than one-third of authors had financial COI affecting 70% of guideline panels.
    -Taylor and Giles 2005. Cash interests taint drug advice

    Pharmaceutical companies have used guidelines to influence physicians prescribing choices and have been considered more effective than using key opinion leaders.
    – Wang, Montori and Murad 2010. Financial conflicts of interest in biomedical research: the need to improve the system.

    3) The majority of evidence is of low quality
    – Tricoci et al. JAMA 2009: Scientific Evidence underlying the ACC/AHA Clinical Practice Guidelines 53 guidelines ~7,200 recommendations with only ~10% being level A with 50% being being level C (consensus, case studies, standard of care)

    4) Guidelines often get reversed
    – DURABILITY OF CLASS I AMERICAN COLLEGE OF CARDIOLOGY/AMERICAN HEART ASSOCIATION CLINICAL PRACTICE GUIDELINE RECOMMENDATIONS Neuman, M.D., et al, JAMA 311(20):2092, May 2014
    – Class 1 American College of Cardiology/American Heart Association clinical Practice guidelines over a median of 6 years. 20% were not retained, downgraded or reversed 9% or omitted 11% in subsequent guidelines

    5) Knowledge Gap
    There is a delay in developing and disseminating guidelines which often result in them being out of date. This contributes to the knowledge translation window that has been quantified at over 10 years.

    POTENTIAL SOLUTIONS:

    1) Emphasize that guidelines are only to “guide” care and not be used by lawyers and administrators to dictate standard of care.

    2) Conflicts of interest (financial and intellectual) need to be minimized. They cannot be financed by industry, patient advocacy groups (back door for industry) and should have methodologists (non-clinicians) on the committee to objectively evaluate the evidence

    3) Improve the quality of evidence through better clinical research. This needs to be done at multiple levels including journals decision editors and peer reviewers.

    4) Reversal – better research is less likely to be reversed, especially if recommendations are conservative

    5) Cut the KT window by making guidelines living documents. Engage social media and crowd source feedback. Update the guidelines in real time as new information becomes published

    These are just a few thoughts. Good luck with your presentation.

  3. Hi Justin,
    Fantastic post and what a great opportunity for all of us to make a difference…my issues with the current guidelines include:

    1. Guidelines are only updated every 3 – 5 years, and yet in the meantime, new evidence is constantly being published. It would be nice to have a system to incorporate new studies into the guidelines (i.e. a living document)
    2. As you have stated the guideline recommendations are made by a select few (Usually in academic positions). Incorporating community providers and low income countries is an essential part of getting a better sense of how best to make recommendations in several systems of practice instead of just in the ivory towers of medicine.
    3. Finally, there are large conflicts of interest in the guidelines, both in funding of research, and the people making the recommendations. It would be nice to have providers who don’t have any of these conflicts of interest as well as blinded statisticians and methodologists to help support or refute the guidelines where needed (i.e. confidence intervals).

    Appreciate you writing this and the opportunity to contribute to medicine on a larger scale.

    Salim

  4. Justin – thank you for this piece. Guidelines, as you say, are ubiquitous throughout medicine but, they are severely flawed. Instead of getting rid of them, we should be improving them and making them more usable.

    First, committee members with conflicts of interest should be removed from guideline panels. There is no role for physicians who have these conflicts to be guiding care on a national or international level. A move like this gives the guidelines more strength as well.

    Second, the committee should contain a group of methodologists/statisticians/researchers to grade the evidence. Preferably, this group should not be actively engaged in work in the area of the guideline. The job of this group is to grade the quality of the evidence devoid of any specific interests in the topic. This move would legitimize the evidence review process.

    Third, guideline committees should be made up of providers from a range of practice settings (academic, community etc) and training backgrounds where relevant (ie paramedics, nurses, advanced practice providers etc). Why should a physician with no pre-hospital training dictate care of pre-hospital providers?

    Fourth, when contentious guidelines are created, active debate should be embraced. Open this up to the community of practice for comment, critique etc. A good example of this is with the ACEP guidelines on thrombolytics in ischemic stroke from a couple of years ago.

    Future guidelines should be free of financial influence and conflicts, embrace true analysis of the evidence, embrace the entire community of practice and, foster open lines of communication with the community.

    1. Thanks for the comment Swami.
      I think one of the huge benefits of involving the social media community in the guideline process is the ability to bring a large number of prospectives to the table. Not everyone can attend the meeting, but everyone can be part of the conversation.
      As a social media expert, how would you suggest making this happen? How does ILCOR actively get our community involved?

  5. One problem with guidelines is that they often seem to exist in their own vacuum. For example, the Infectious Diseases Society of America recently released a guideline on the management of C. Difficile that includes a change of first-line treatment from metronidazole to oral vancomycin. This is reasonable based on resistance patterns. But what we don’t know is to what extent this change is a result of other previous and seemingly (but not) unrelated guidelines that fed into these resistance patterns. For example, guidelines by other groups have advocated for far lower standards for administering broad spectrum antibiotics than the data suggest are necessary. Have guidelines like that led to more C. Difficile infections which in turn become more resistant? So, one area that guidelines need to improve on is for authors to acknowledge the idea that their work could positively or negatively effect patients in total, and not just to think about the disease process in question.

    Another problem with guidelines has been authorship. Content experts are frequently biased and have either financial or career-oriented conflicts of interest. Meanwhile, methodologists are frequently removed from reality. So who should be writing these? Is a bad guideline better than no guideline? I would say yes, in many cases. As I wrote with Scott Weingart in 2014 (https://onlinelibrary.wiley.com/doi/full/10.1111/1742-6723.12192), one helpful adjunct would be engaging thought leaders via social media. These voices are often highly educated and involved, and without conflicts of interest nor the cool distance of not practicing medicine. Sometimes all it takes to point out that a guideline has major problems is for it to simply be available for public comment (which must be more than just circulating a draft among the inner-circle of a particular field) and taking the comments from those opportunities seriously.

    1. Thanks for the comment!
      The use of thought leaders as part of a process of post-publication peer review is brilliant, and definitely something we are seeing a lot more of as a result of social media. Right now, that process is relatively ad hoc. Any thoughts on how an organisation like ILCOR can harness the expertise within the social media community in a more organised way? Is it good enough that this occurs after the guidelines have been published, or should we be striving for a more integrated approach right from the outset?

  6. Hey Justin, great post.

    I’ve been thinking about this problem a lot recently. I came across the idea that a really scalable, low cost, easy access way to attack this task would be to borrow the Wiki concept.

    We could make the entire ILCOR guideline a wiki that would allow everyone in the FOAMED world to propose and justify changes as well as to propose new topics to be included.

    A Wiki has the important feature of providing a change log that can inform practitioners of how a recommendation came to be included. And I think you will agree that Wikipedia has proven that this format is scalable.

    Of course the key is in who is appointed as editors. And the protocols regarding how and when changes are accepted needs to be carefully constructed so that nuance and uncertainty is not averaged over. But the wiki approach does provide a tractable framework for documenting the editors decisions in a transparent manner.

    1. Thanks for the comment Sean
      I agree completely. Somewhere in my stash of half written blog posts, there is an essay on this exact idea. Wiki is a proven, existing platform that makes a lot of sense for guidelines. Two issues I haven’t totally sorted out yet. 1) science isn’t a democracy. Although I think that everyone should have a voice, sometimes there is a most appropriate answer, and it isn’t clear that the wiki format necessary leads to that answer. 2) who is allowed to edit? I think it is a mistake to limit contributions to MDs, but at the same time medical guidelines still require some expertise, and probably shouldn’t be open to just anyone.

  7. Justin:

    Thank you for posting these ideas and opinions about guidelines on your blog. I agree with these perspectives as far as increasing the range of voices including writing guidelines (including rural and non-academic voices), ensuring that guideline authors are free of compromising influences or conflicts, and optimizing transparency in guideline development. Social Media definitely adds a “new” opportunity to includes a broader range of stakeholder voices (more front-line clinicians, various specialties, patients) and accelerate awareness of guidelines and guideline updates. I do have two challenges to your rant, though.

    First, how are social media audiences supposed to know whether social media voices are conflict free? There are no social media publishing guidelines or standards necessitating declaration of any form of conflict of interest? On a related thought, how are social medial audiences supposed to judge the validity of the popular voices’ opinions or declarations since GRADE-like criteria for podcasts and blogs really don’t exist and (as far as I know) most podcasts and blogs don’t have any identifiable rating with something like the METRIQ score.

    Second, the essay neglects another call to arms action for your audience: join the ACEP Clinical Policy Committee, ILCOR, or other specialty guideline-writing bodies. Admittedly, this isn’t as easy as logging onto a website and expressing interest or picking up the phone to call ACEP headquarters – but joining these guideline development teams is feasible with effort and persistence. Additionally, other emergency medicine organizations may someday begin generating guidelines and open additional opportunities to contribute at all stages of guideline development.

    Good luck when you present at ILCOR. This blog post should encourage guideline developers to think outside the box, using disruptive innovation and thought-leaders like yourself to make future guidelines more meaningful and less distasteful for conscientious physicians on the front-line.

    Chris Carpenter

    1. Thank you so much for the comment Chris.
      I agree I was only focused on the one call to action (one thing at a time), although a lot of my suggested solutions might also be seen as calls to action.
      The question of conflict is an important and difficult one. There is so much conflict in our current guidelines, I am not sure that conflict among added voices will be the most important issue, but in the long run it is incredibly important. We need patient voices in medicine, but my understanding is that a lot of patient advocacy groups are really just fronts for industry. The FOAMed community as it exists now is a pretty wonderful example, but as soon as someone sees a profit to be made, I am sure we will see that change. I would love to hear other people’s suggestions on how to deal with conflicts as we encourage engagement from a larger community. In my mind, the ideal world is one where doctors don’t have financial conflicts, so we can just stop worrying about disclosures.

  8. Guidelines. I love them AND detest them. I think there are a few issues here:
    1. Guidelines are often great interpretations of the literature by “experts” but there may be biases here because these “experts” authored many of the studies or may take $ from industry. Further, many of these experts are removed from daily clinical practice or have monolithic practice (ex: academic tertiary care setting).
    2. Guidelines SHOULD have stakeholder involvement. Sure, I’m sure there’s some polling that goes on behind closed doors but any good dissemination & implementation strategy needs to have true stakeholder involvement. Social media offers an interesting way to make this happen …to solicit feedback (not necessarily given via social media but the invitation to give feedback). A public draft and response period with true, targeted attempts to get feedback from a large swath (from Social Media and movers and shakers in each field that weren’t on the committee) could improve the transparency of this process.
    3. Guideline revisitation – once a guideline is written things shouldn’t be finished. There should be a way for individuals to give feedback. Evidence is not static.

Leave a Reply