Statistical robustness of randomized controlled trials in high-impact journals has improved but was low across medical specialties

Objectives: To determine whether the statistical fragility of randomized controlled trials (RCTs) in high-impact journals has improved in the last decade and to perform an umbrella review of all published data on the Fragility Index (FI) across medical specialties. Study Design and Setting: The FI was calculated for all eligible RCTs published from 2014–2021 in the New England Journal of Medicine, The Lancet, the Journal of the American Medical Association, the British Medical Journal, and the Annals of Internal Medicine. Trials reporting dichotomous, statistically significant, superiority results were eligible. All previously published systematic reviews on the FI were included in the umbrella review and analyzed by medical (sub) specialty. Results: Of 2,544 screened RCTs, 643 were eligible for the FI analysis. These had a median sample size of 625 (interquartile range [IQR]: 265–2,056), a median FI of 12 (IQR: 3–28), and a median Fragility Quotient of 0.015 (IQR: 0.004–0.045). This is an improvement compared with the median FI of 8 (IQR: 3–18) of RCTs published a decade earlier in the same five journals (P < 0.001). The umbrella review included 57 publications across 15 different medical specialties, with a total of between 10 and 692 RCTs for each specialty. The median FI ranged between two and four for all disciplines. Conclusion: In the last decade, the median statistical robustness of RCTs published in high-impact journals has improved, yet the unchanged lower bound of the interquartile range reveals that statistical significance in 25% of trials is still dependent on three or less events. The umbrella review revealed that statistical fragility is prevalent across all medical specialties. The FI is an easy-to-understand metric that can be used to supplement reported P values and help readers look beyond merely reaching statistical significance.