Observational Research for Therapies Titrated to Effect and Associated With Severity of Illness: Misleading Results From Commonly Used Statistical Methods

OBJECTIVES: In critically ill patients, treatment dose or intensity is often related to severity of illness and mortality risk, whereas overtreatment or undertreatment (relative to the individual need) may further increase the odds of death. We aimed to investigate how these relationships affect the results of common statistical methods used in observational studies. DESIGN: Using Monte Carlo simulation, we generated data for 5,000 patients with a treatment dose related to the pretreatment mortality risk but with randomly distributed overtreatment or undertreatment. Significant overtreatment or undertreatment (relative to the optimal dose) further increased the mortality risk. A prognostic score that reflects the mortality risk and an outcome of death or survival was then generated. The study was analyzed: 1) using logistic regression to estimate the effect of treatment dose on outcome while controlling for prognostic score and 2) using propensity score matching and inverse probability weighting of the effect of high treatment dose on outcome. The data generation and analyses were repeated 1,500 times over sample sizes between 200 and 30,000 patients, with an increasing accuracy of the prognostic score and with different underlying assumptions. SETTING: Computer-simulated studies. MEASUREMENTS AND MAIN RESULTS: In the simulated 5,000-patient observational study, higher treatment dose was found to be associated with increased odds of death (p = 0.00001) while controlling for the prognostic score with logistic regression. Propensity-matched analysis led to similar results. Larger sample sizes led to equally biased estimates with narrower CIs. A perfect risk predictor negated the bias only under artificially perfect assumptions. CONCLUSIONS: When a treatment dose is associated with severity of illness and should be dosed "enough," logistic regression, propensity score matching, and inverse probability weighting to adjust for confounding by severity of illness lead to biased results. Larger sample sizes lead to more precisely wrong estimates.