Mirror Mirror on the Wall, Who’s the Most Fragile of Them All? Assessing the Fragility Index of ECASS III is a guest post by Kyle DeWitt
There are two kinds of people in the emergency department: those who are advocates of intravenous tPA for the treatment of acute ischemic stroke, and those who aren’t. Among twelve large clinical trials, only two have demonstrated statistically beneficial results of systemic thrombolytics in the setting of acute ischemic stroke: NINDS and ECASS III [1,2,3]. Despite the lack of robust high quality evidence, current American Heart Association/American Stroke Association guidelines recommend the administration of intravenous tissue plasminogen activator (tPA) as the “standard of care” for treatment of acute ischemic stroke .
Josh Farkas (@PulmCrit) recently posted a brilliant reanalysis of the NINDS trial using a metric known as Fragility Index . Fragility Index measures the statistical reproducibility of a trial by determining the minimal number of patient outcomes that must be changed in order to shift the p-value above 0.05. The smaller the Fragility Index, the more fragile the study. Although this concept is relatively new, it has already been used to evaluate a large number of trials in the critical care setting .
The NINDS trial was found to have an overall Fragility Index of three, meaning that if three additional patients in the control group were to have a favorable outcome the study would have shown no statistically significant difference between tPA and placebo at 90 days. Frightening, right?
Similar to NINDS, ECASS III is fraught with limitations. Both trials were poorly randomized with an imbalanced, sicker (greater NIHSS score) placebo group. Furthermore, the primary end point of disability at 90 days was dichotomized as “favorable” for modified Rankin scale (mRS) scores of 0-1 or “unfavorable” for mRS scores between 2-6. So based on this, a mRS score of 2 (slight disability, still independent) is considered the same outcome as a mRS score of 6 (dead). This is rubbish. Not surprisingly, the beneficial signal of tPA is lost when patients with mRS scores between 0-2 at 90 days are considered favorable. Given these limitations, the Fragility Index of ECASS III should be evaluated to determine if the results are more reproducible than the NINDS trial.
Calculating the Fragility Index of ECASS III
ECASS III assessed the efficacy and safety of tPA administered between 3 and 4.5 hours after the onset of stroke symptoms and provided the rationale for the “extended treatment window” in current practice guidelines. The authors concluded that more patients had a favorable outcome with alteplase vs. placebo (52.4% vs. 45.2%; OR, 1.34; 95% CI 1.02-1.76, P = 0.04). These results are depicted in the table below.
To determine the Fragility Index of the primary endpoint, contingency tables were created to move one patient at a time in the control group from an unfavorable outcome to a favorable outcome. After each data point manipulation, subsequent p-values were recalculated with Fisher’s exact test until the p-value reached 0.05. Figures were checked for accuracy with the use of an online Fragility Index calculator . The result was almost unbelievable: one. If only one additional patient in the control group were to have a favorable outcome the study would be considered negative. [The Fragility Index of the primary outcome is increased to eight when data from the per-protocol analysis is used. This larger value is expected given that per-protocol analyses are subject to bias and often favor the intervention arm.]
A favorable “global outcome” with alteplase compared to placebo was also reported as a secondary end point. This was the composite of several individual outcome scales which included mRS, Barthel Index, NIHSS, and Glasgow Outcome Scale (see table above). The positive global outcome signal is primarily driven by statistically significant differences in mRS and NIHSS scores. By simply using Fisher’s exact test to produce an exact p-value rather than an estimate for differences in NIHSS scores, the p-value is then shifted to 0.0501. This would result in a Fragility Index of zero, extremely unlikely to be reproducible.
Number of “favorable outcomes” added to control group
mRS score 0-1
NIHSS score 0-1
Intracranial hemorrhage and symptomatic intracranial hemorrhage (sICH) rates were associated with larger Fragility Indices of fourteen and three, respectively. This provides further validation that the detrimental effects of tPA are much more reproducible compared to those of it’s efficacy.
In the meantime, intravenous tPA will continue to remain the “standard of care” for up to 4.5 hours after stroke onset despite overwhelming evidence of no benefit or harm. The American College of Emergency Physicians have at least downgraded the strength of this recommendation:
“Despite the known risk of sICH and the variability in the degree of benefit in functional outcomes, IV tPA may be offered and may be given to carefully selected patients with acute ischemic stroke within 3 to 4.5 hours after symptom onset at institutions where systems are in place to safely administer the medication.” 
The glass remains half empty…
Take home points
- Fragility Index measures the statistical reproducibility of study outcomes. A low Fragility Index indicates less statistically robust results.
- ECASS III has a Fragility Index of one, meaning it would only take one additional favorable outcome in the control group to render a nonsignificant difference between tPA and placebo.
- Similar to the NINDS trial, ECASS III has an extremely low Fragility Index which suggests that the “beneficial” results of these studies are likely not reproducible.
More from EM PharmD related to Assessing the Fragility Index of ECASS III:
Reverse anticoagulation for rt-PA in stroke
Atrial Fibrillation Management and Drug Therapy
- Tissue plasminogen activator for acute ischemic stroke. The National Institute of Neurological Disorders and Stroke rt-PA Stroke Study Group. N Engl J Med 1995;333(24):1581-7.
- Thrombolysis with Alteplase 3 to 4.5 Hours after Acute Ischemic Stroke. N Engl J Med 2008;359:1317-29.
- http://www.thennt.com/nnt/thrombolytics-for-stroke/. Accessed electronically August 2, 2016.
- Guidelines for the Early Management of Patients With Acute Ischemic Stroke. Stroke 2013;44:870-947
- http://emcrit.org/pulmcrit/fragility-index-ninds/. Accessed electronically August 2, 2016.
- Ridgeon EE, Young PJ, Bellomo R, et all. The Fragility Index in Multicenter Randomized Controlled Critical Care Trials. Crit Care Med 016;44(7):1278-84.
- http://www.fragilityindex.com. Accessed electronically August 2, 2016.
- Clinical Policy: Use of Intravenous Tissue Plasminogen Activator for the Management of Acute Ischemic Stroke in the Emergency Department. Ann Emerg Med 2015;66:322-333.
- Assessing the Fragility Index of ECASS III