Lessons from the pandemic: Lagging indicators

The time has come to learn from the pandemic; to learn from our mistakes (and hopefully a few successes). There are many topics to choose from. In future posts, I plan to be very positive about the tremendous work done by so many colleagues. I will probably also have to write about our incredible scientific failures, and the fact that we clearly revealed that a medical degree does not imply scientific understanding or critical thinking ability. However, I want to start by reflecting on a topic that most of us are likely to forget, or at least not extrapolate beyond the lessons we learned in infectious disease: lagging indicators. 

If you are like me, you probably never heard the term “lagging indicators” before the pandemic. However, the concept suddenly became very important in understanding the world around us. It became clear that the consequences of our actions weren’t immediately apparent. The COVID measures that were most important – mortality, ICU admissions, and hospital admissions – were all lagging indicators. They did not accurately reflect the current state of the world, nor the current need for intervention. 

Unfortunately, over and over again, we saw decisions made that ignored that fact that these indicators lagged. Despite massive spikes in cases, making spikes in hospitalizations inevitable, we were told that no public health interventions were needed because hospitalizations were not yet rising. Time and time again we waited too long to intervene because we were using metrics that represented the state of the world weeks or even months ago, without recognizing that time lag.  By the time hospitalizations started to rise, it was too late to avert the disaster.

I don’t think our decision makers understand lagging indicators to this day, but I wanted to move beyond COVID and reflect on the implications of this concept for other areas of our lives. 

The concept that test results might “live in the past” is important, but relatively well recognized in emergency medicine. In an actively bleeding patient, we know not to base our management on hemoglobin levels, because they don’t accurately represent the current disease state. Similarly, in shock states, we recognize that it might take many minutes for blood to reach the extremities, and therefore the oxygen saturation on the monitor might represent a physiologic state from many minutes in the past. 

Lagging indicators might also help people better understand personal health. A good physique is a lagging indicator of healthy habits, both in terms of eating and exercise. If you only focus on your physique, like we only focused on hospitalizations during COVID, your approach to health is likely to be misguided, because you won’t see the consequences of your actions for weeks to months.

Good financial decisions are also seemingly driven primarily by lagging indicators. A large bank account is a lagging indicator of sound financial policies. It takes time and discipline to save money and invest it soundly. A focus on day to day financial gains is a recipe for financial disaster. It is likely to lead to bad decisions and overreactions. In fact, many financial advisors suggest never looking at your investment portfolio, but instead focusing on a fundamentally sound strategy, because the success of a plan will not become apparent for decades. 

The realizations that many important outcomes are long term and lagging means we need to focus our attention elsewhere. There seem to be two primary alternatives. We can identify short term surrogates that accurately reflect the outcomes we truly care about, or we can simply focus on the process and trust the outcomes will follow. Both have benefits, but also pitfalls we should try to avoid.

In the context of evidence based medicine, we have discussed surrogate outcomes many times before, and often I am skeptical of studies that overuse them, because surrogates can easily mislead us. A decrease in cholesterol or blood pressure does not necessarily lead to a better or longer life, so we need to be cautious in over-interpreting such results. However, surrogates are often important measures of good process. Emergency physicians should be aware of their first pass success rate when intubating, because it is a good surrogate, in that it has a strong association with positive outcomes for patients. However, even first pass success has its limitations. Two controlled intubation attempts, where a problem was quickly identified on the first and corrected with a change in technique without hypoxia, is clearly better than a single prolonged attempt where the tube is blindly shoved in the vague direction of the trachea as the oxygen saturation begins to fall.

COVID taught us to be flexible and creative in considering the most important indicators. Sometimes, no indicators exist. With COVID, we literally had to invent a test to monitor sewage for early cases. We also learned to think tangentially. For example, measuring carbon dioxide turned out to be a great (although underused) surrogate indicator of air quality and therefore COVID transmission risk in indoor spaces. However, neither of these surrogates were close to perfect, and also act as an important reminder of the need to constantly reevaluate whether the surrogates we are using are accurately reflecting the outcomes we care most about.

Despite our love of tests and outcome measures, I think we need to recognize that we usually won’t have an adequate surrogate outcome, and that we will need to focus on process measures instead. Even when we have good tests, success can render those tests unhelpful. For example, if we had ever been truly successful at managing COVID and kept community infection rates at very low levels, none of our tests would have been very helpful, because so few people would have tested positive. The lack of positive COVID tests would have been a marker of the success of our strategies, and shouldn’t be interpreted (as they often were) as an indication to stop using those strategies. In the ideal world, we would have continued to focus on low cost strategies, such as masking in public, immunization, outdoor gatherings when possible, and good ventilation, and kept COVID levels constantly low without any significant impact on people’s lives. Instead, we ignored those process measures, focused on lagging indicators, and the result was significant harm because we allowed matters to get out of control. 

Focus on process instead of waiting for markers of failure. I have auto-deposits set up to save for retirement, rather than waiting for some indication that I have enough money. I do a basic workout every day, without worrying about whether I am going to look good at the beach this year. I continue the process of daily medical education, despite the fact I no longer take tests or exams to measure the success of that process. 

I don’t think we have accepted these lessons in medicine. Emergency departments are frequently judged by their wait times, but the wait to see an emergency physician is actually a lagging indicator. If 50 patients all arrive at the emergency department at the same time, the wait for the first patient might only be 5 minutes, but the short wait masks a clear resource problem. It is impossible for there to be a 3 hour wait until 3 hours pass, but if you let 3 hours pass without acknowledging the massive influx in demand, you are way behind the game. The time to call for help is when the massive influx of patients arrives, not when the computer system flags longer wait times hours later. 

How do you measure success? What indicators are you using? Are there better options?

---

Read more on the Rants and Ramblings section of First10EM.