The PECARN c-spine rule could cause a lot of harm

As with everything that comes out of the PECARN group, this is a huge study, and is therefore one that everyone needs to know about. That being said, despite working in two very busy community hospitals with a high percentage of pediatrics visits, c-spine injuries are just not an issue I struggle with. I have seen 2 pediatric c-spine injuries in over a decade of work, and both were incredibly obvious. (In both cases, the kids were holding their heads still with their hands, and said something along the lines of “if I let go, my head is going to fall off’.) Considering the major problems with the science base for our current decision rules, and the problems of using high sensitivity low specificity tests in populations with incredibly low pretest probabilities, I started reading this paper with significant skepticism. On the other hand, the Canadian c-spine rule is one of the very few decision rules with a properly controlled implementation study demonstrating benefit, and the question of whether the c-spine is broken is simple enough, so c-spine injuries are one area we might expect decision rules to be successful. So what does the trial say?

The paper

Leonard JC, Harding M, Cook LJ, Leonard JR, Adelgais KM, Ahmad FA, Browne LR, Burger RK, Chaudhari PP, Corwin DJ, Glomb NW, Lee LK, Owusu-Ansah S, Riney LC, Rogers AJ, Rubalcava DM, Sapien RE, Szadkowski MA, Tzimenatos L, Ward CE, Yen K, Kuppermann N. PECARN prediction rule for cervical spine imaging of children presenting to the emergency department with blunt trauma: a multicentre prospective observational study. Lancet Child Adolesc Health. 2024 Jul;8(7):482-490. doi: 10.1016/S2352-4642(24)00104-4. Epub 2024 Jun 4. PMID: 38843852 NCT05049330

The Methods

This is a large multicentere prospective observational trial with both a derivation and validation cohort from level 1 pediatric trauma centres in the United States. 

Patients

Children 0-17 years of age presenting to a level-1 pediatric trauma center with known or suspected blunt trauma and one of:

• Evaluation by a trauma team
• Transported from the scene of injury to the participating hospital by emergency medical services
• Underwent cervical spine imaging at the participating hospital
• Underwent cervical spine imaging before transfer to the participating hospital

Intervention

They used a 2 step process to derive the decision rule in the first cohort, and then validate the rule in the second. 

Outcome

The primary outcome was cervical spine injury, defined as fractures or ligamentous injuries of the cervical spine, cervical intraspinal hemorrhage, vertebral artery injury, cervical spinal cord injury (including SCIWRA). This was determined by medical record review or contact with the patient’s surgeon or parents at 21-28 days. 

The Results

The derivation cohort screened 26,935 children, 16,206 of whom were eligible, and 11,857 were enrolled. They don’t seem to explain why so many eligible patients were not enrolled, which is a problem.

The validation cohort screened 23,320 children, 16,061 of whom were eligible, and 10,573 were enrolled. Again, this is a very high percentage of eligible patients who are just not accounted for.

They identify 4 clinical factors that are associated with a very high risk of c-spine injury, and are clearly important: GCS 3-8, AVPU scale “unresponsive”, abnormal airway, breathing, or circulation, and focal neurologic deficit.

These are very high risk factors, representing at least a 10% risk of c-spine injury, and are therefore important to know. Unfortunately, there were still 156 c-spine injuries in the 10,929 children who didn’t have any of these features. This represents a 1.4% risk, and I think a lot of the future conversation around this paper will focus on whether the additional steps in this rule is worth the effort and harm to get this 1.4% lower.

To catch most of these remaining injuries, they added 5 additional variables: neck pain, altered mental status (GCS 9-14, AVPU “verbal or pain”, or other signs of altered mental status), substantial torso injury, substantial head injury, and posterior midline neck tenderness. As you can see from the results, these factors are far less helpful. 

The result is a rule that suggests CT in the high risk population, x-ray in the middle risk population, and no imaging in the low risk population. The resulting 9 factor decision tool was then tested in the validation cohort, which had a prevalence of c-spine injury of 1.5%. The rule had a sensitivity of 94.3% (95% CI 90,7-97.9) and a specificity of 60.4% (95% CI 59.4-61.3). (The sensitivity and specificity were actually a little worse in the derivation cohort.) Although they emphasize the negative predictive value of 99.9%, you can’t talk about that without also emphasizing the abysmal positive predictive value of 3.5%!!

They don’t present the likelihood ratios, but I calculate them as a positive likelihood ratio of 2.35 and a negative likelihood ratio of 0.1. If you used this test in 1,000 children, 15 would have a c-spine injury and the rule would catch 14. Unfortunately, out of the 985 well children, this rule would be false positive and suggest imaging in 394!

My thoughts

This is a tremendous undertaking, and an incredibly valuable piece of science. Every emergency physician should be familiar with this paper. However, the rule should absolutely not be used at this time, at least in most emergency departments around the world. Unfortunately, I worry the horse is out of the stable, and this rule might end up hurting a lot of children.

There are the usual decision rule cautions. This hasn’t been externally validated, and decision rules very frequently look a lot worse when retested in external populations. This hasn’t been tested against the gold standard of physician judgement. We don’t have an implementation study, so have no idea if this would decrease (or actually increase) imaging in real life practice. However, there are a lot of reasons to think this rule could end up increasing imaging a lot, and cause a lot of harm. 

First of all, this cohort represents an incredibly high risk population presenting to level 1 trauma centers. Even in this very high risk population, the baseline risk of c-spine injury was only about 1%. For the average child presenting with much less severe trauma to the average emergency department, the baseline risk will be much lower, and the false positives will absolutely swamp the true positives. 

Let’s imagine that the risk for patients presenting in the community is 10 times lower than for those presenting as trauma team activations. (I think it is probably much lower than that.) That would mean in a population of 1,000 patient, there would be one c-spine injury, which this rule would probably catch, but the cost would be sending 400 child for imaging, the vast majority of who would never have been imaged prior to the publication of this study. (This rule would probably require me to image hundreds of kids a year, whereas my current rate is a handful over more than a decade of practice.)

The authors claim that this rule would decrease imaging in their centers, but that is based on a baseline where almost 60% of children are being imaged, including 17% receiving CT. I hope that highlights how different this population is to the population we see, where c-spine imaging rates are way below 1%. It also highlights an important bias in this study design, in that one of the key inclusion criteria for the study was “c-spine imaging was done”. Specifically including just the subset of patients in whom a clinician thought imaging was required biases the population, and means we should not expect the rule to behave the same in a general pediatric emergency population.

The biggest problem with this study is that they forgot to include a control group. Without comparing the rule to the current gold standard of physician judgment, we have absolutely no idea whether this will make practice better or worse. Considering that this is mostly a non-issue around the world, and pediatric c-spine imaging is ordered very rarely in the average emergency department, it seems like it will almost certainly make practice worse.

This is a common problem with decision rules. They are almost never compared to physician judgment, and when they are, they almost never win. (Sanders 2015, Schriger 2017) For example, the PECARN head injury rule, which has very similar test characteristics to this rule, is substantially worse than clinical judgement. Clinical judgement will catch all of the important head injuries, but using the PECARN rule will result in a substantial increase in imaging (unless you work in a place with a ridiculous baseline imaging rate). (Babl 2018)

Unfortunately, unlike some previous excellent decision rule work (the PERC rule), this study doesn’t not include a calculation of the test threshold. That is very problematic for this type of research, because it means you have no target, and no standard against which to judge the validations. A 0% miss rate is impossible, and any rule that targets that level will cause more harm than good. An undefined miss rate is even more problematic, because you have no objective criterion against which you can measure success. The sensitivity here was below 95%. Is that good enough? The negative predictive value here is excellent, but that is population dependent. How will we judge the necessary external validations? If future a validation shows a miss rate of 0.5%, is that good enough? 1%? 2%? Because they didn’t start by calculating the test threshold, we just don’t know. What we do know (and they outline very well in the introduction) is that there is a lot of harm from imaging children’s necks, and so there is definitely a test threshold that needs to be considered. It just isn’t clear how this rule compares. 

The Hawthorne effect is also always important to consider in these studies. These children didn’t get a ‘normal’ examination. They received an examination by a physician that had a research checklist that needed to be filled out, and therefore their assessment could differ significantly from times when clinicians are not prompted with that research checklist. (Or when their actions are not being observed by research assistants.)

I don’t understand why they missed more than 30% of eligible patients, and it isn’t well explained in the manuscript. There do seem to be some clinical difference in the patients missed (higher admission rate but fewer arrived by EMS), and if these patients were systematically different in some way, this could represent a major source of bias in this data.

It is important enough to mention it again: decision rules should not be used after a single validation, even if the validation is huge. We have a long history of rules that look substantially worse in external validations, and a validation only gives you test characteristics, but tells you nothing about the comparison to current practice, or whether the rule actually results in net benefit or harm. 

Even without those limitations, a rule with a sensitivity of 94% (and a 95% CI down to 90%) does not seem good enough for a high risk condition, and the low specificity means that this will be the D-dimer of trauma. If it is used indiscriminately it will cause great harm. Used with caution in only those patients with this highest risk (i.e. those at a pediatric trauma center), this rule might provide value, although at this point, that value is unproven. 

CT versus x-ray

This rule takes the interesting step of suggesting CT for high risk patients and x-ray for low risk patients. In principle, I like this idea. However, it must be pointed out that that recommendation is not at all science based. This study did not test the rule, as written, at all. They provide no data to prove that x-ray is good enough in the lower risk population, nor, for that matter, that CT is actually required in the higher risk population. For the purposes of this article, I am not going to do a systematic review of the accuracy of c-spine imaging modalities in a pediatric population. You can make some estimates yourself based on the pretest probabilities of the 2 groups described and the negative likelihood ratios of the two tests. However, without a discussion of the test threshold, it is next to impossible to know which test is correct, because we don’t know our target post-test probability. 

A brief thought experiment

When I read this study, I am almost 100% sure that it will increase c-spine imagining dramatically in the pediatric population. Obviously, I cannot prove that, and we really need to see some implementation studies to determine the rule’s impact on practice, but I think with a simple thought experiment, you can convince yourself I am correct.

Let’s imagine a world in which this rule is used perfectly. This world won’t actually exist. We will see indication creep. This rule will be used widely, with little regard for pretest probability. This rule will be applied incorrectly. But let’s ignore all that and imagine a perfect use case. In this scenario, the rule is only applied after the imaging has already been ordered and the patient is actually sitting in radiology to get their test. In this setting, the rule cannot possibly increase imaging, because the test is already ordered, and so can only possible decrease testing.

If you take a step back, what does this rule say? It says that you can cancel that imaging order if: the patient is not altered, required no intervention for their ABCs, has no focal neurologic deficits, has no neck pain, has no substantial head or torso injury, and has no tenderness on examination. Think about that patient. In your current practice, would you ever order imaging on that patient? Why in the world would you order imaging on a patient with no pain, no tenderness, no neurologic findings, and no other substantial injuries? That makes no sense at all. No clinician I have ever worked with would ever order imaging in that patient.

We sometimes get lost in the science jargon of a paper and lose track of common sense. Forget about the numbers they present and just think about what the rule is saying. No emergency physician would ever order c-spine imagine in an awake patient with no pain, no tenderness, no neurologic deficits, and no other substantial injuries. (Or, if you were ordering imaging, you probably had some other really good reason, and it isn’t clear a rule with a sensitivity of 95% should talk you out of that test.) If you would never order imaging in that patient, there is no possible way this rule could decrease imaging in your practice. 

I will say that again: there is no way this rule could possibly reduce imaging, because no sane emergency physician would ever order imaging in the low risk patient being described. You don’t need this rule, because you are already excluding these patients using basic clinical judgement.

However, we know the rule will not be applied in the perfect scenario this thought experiment describes. It won’t be applied only after imaging has been ordered. It will be applied in everyone. It will be applied in patients who you currently would never even consider imaging. You will get worried about the exact definition of substantial head injury, or you will waffle about the fact the kid might have said his neck hurt 3 hours ago on scene despite having no complaints now, or you will wonder whether a child’s natural aversion to being touched was actually tenderness. You will worry others will judge you in the case of a miss if there is an objective rule they can point to that says you should have imaged. And so you will image when you wouldn’t previously have imaged.

In some clinical scenarios, increased imaging might be acceptable, if it resulted in fewer missed injuries. Pediatric c-spine injuries is not one of those scenarios. One thing I can tell you for certain: there is not an epidemic of missed cervical spine injuries. We don’t need help making this diagnosis. We aren’t missing these children. So any increased imaging will almost certainly represent pure harm. 

We need implementation studies to be sure, but I think this thought experiment makes it very clear that this decision rule will dramatically increase the use of c-spine imaging in children, likely resulting in tremendous harm.

I can say with absolute certainty this rule would increase imaging in my practice. It would increase imaging in every department I have ever worked in. It would definitely increase imaging in the Countries I have worked in (Canada, Australia, and New Zealand). I think it would increase imaging at essentially every community emergency department in the world, because if not we have some dramatic issues with over-testing. Perhaps the biggest issue we face in the modern era of decision rules is that so many of these rules are being produced in a country that is a dramatic outlier in almost all areas of clinical medicine.

A few subtle points about using the rule

Although I don’t think this rule should be used, I know people are going to use it, and so it is important to know what the rule is actually saying.

Aside from the advice to use the rule, there was an excellent EM:RAP segment on this paper, which includes an interview with the lead author who made a number of statements that were not completely clear to me from the manuscript.

In the manuscript they state that “substantial injuries” of the head or torso are those that required surgical intervention or inpatient observation. I expect those criteria will get expanded in real practice, especially with the uncertainty and variations in practice around which injuries need inpatient observation. However, at least there was a clear definition.

What counted as “abnormal airway, breathing, or circulation” was unclear to me when reading this paper, and I imagine it will be widely misinterpreted. This is especially true in the pediatric population, where even children with trivial injuries present with significant tachycardia due to fear and crying. However, in the audio program, the author made it clear that injuries only counted as abnormal airway, breathing, or circulation if they required you to stop your trauma assessment and intervene. 

With regard to neurologic deficits, reported deficits from before hospital arrival were not counted. Only deficits present during the physician examination counted for the rule. 

For “neck pain”, this was only counted if the child stated that their neck hurt without prompting. You could ask if they had pain anywhere, and if they said their neck hurt that would count. However, you were specifically not allowed to ask “does your neck hurt?”. 

I think these are important, but subtle distinctions, and they make me think that (much like other rules) this rule will be mis-used and mis-interpreted frequently when used outside the research setting, and therefore we can expect to see even worse results in the real world setting.

Bottom line

This is very high quality research resulting the in the derivation and initial validation of a pediatric c-spine rule in a population of extremely high risk pediatric trauma patients.

This rule is not ready to be used clinically. If you happen to work in a pediatric trauma center, and you are seeing high risk trauma patients as part of a trauma activation, and you are considering ordering a pan-scan and want to use this rule to talk yourself out of ordering the c-spine portion of the imaging, I won’t argue with you, although the rule is still a long ways from being proven helpful in even that situation.

However, if you are working in any other setting, this rule will almost certainly increase your use of imaging. It was not studied in your population. It has not been subjected to the necessary external validations. It has never been compared to the current gold standard (clinical judgement). We have no implementation studies. It absolutely should not be used at this time. 

Other FOAMed

Decision rules are ruining medicine

Evidence based medicine is easy

The EBM bibliography

Evidence based medicine resources

EBM deep dives

References

Babl FE, Oakley E, Dalziel SR, et al. Accuracy of Clinician Practice Compared With Three Head Injury Decision Rules in Children: A Prospective Cohort Study. Annals of emergency medicine. 2018; 71(6):703-710. PMID: 29452747

Sanders S, Doust J, Glasziou P. A systematic review of studies comparing diagnostic clinical prediction rules with clinical judgement. PLoS One. 2015 Jun 3;10(6):e0128233. doi: 10.1371/journal.pone.0128233. PMID: 26039538

Schriger DL, Elder JW, Cooper RJ. Structured Clinical Decision Aids Are Seldom Compared With Subjective Physician Judgment, and Are Seldom Superior. Ann Emerg Med. 2017 Sep;70(3):338-344.e3. doi: 10.1016/j.annemergmed.2016.12.004. Epub 2017 Feb 24. PMID: 28238497