Letter to the Editor: Fair Comparisons in the Arena of Risk Scores

TY - JOUR

T1 - Letter to the Editor: Fair Comparisons in the Arena of Risk Scores

AU - Wang, Junfeng

AU - Geskus, Ronald B.

AU - Leeflang, Mariska M. G.

AU - Ponsioen, Cyriel Y.

N1 - Funding Information: Junfeng Wang Ph.D. Ronald B. Geskus Ph.D. Mariska M.G. Leeflang Ph.D. Cyriel Y. Ponsioen M.D., Ph.D. Department of Biomedical Data Sciences Leiden University Medical Center Leiden the Netherlands Oxford University Clinical Research Unit Ho Chi Minh City Vietnam Nuffield Department of Medicine University of Oxford Oxford United Kingdom Department of Clinical Epidemiology Biostatistics and Bioinformatics Amsterdam Public Health Amsterdam UMC University of Amsterdam Amsterdam the Netherlands Department of Gastroenterology & Hepatology Amsterdam University Medical Center Location AMC Amsterdam the Netherlands Wellcome Trust 106680/Z/14/Z Supported by the Wellcome Trust (grant number 106680/Z/14/Z). Potential conflict of interest: Dr. Ponsioen consults for, is on the speakers’ bureau for, and received grants from Takeda. He consults for Pliant and Shire and is on the speakers’ bureaus for Tillotts and AbbVie. To the Editor: We are happy to see several research groups are working on this topic, underscoring the importance of the need for predicting outcomes of primary sclerosing cholangitis (PSC). The authors mention the Amsterdam‐Oxford PSC score, which contains similar variables and evaluates almost the same outcomes, and state that it has a moderate C statistic of 0.68 and suggest the UK‐PSC score is superior due to its C statistic of 0.80. We congratulate Goode and colleagues on their work. However, a C statistic is not an inherent feature of a model: it also depends on the features of the cohort, e.g., censoring distribution, correlation between predictors and outcome, even correlation between predictors; these can all influence the best possible C statistic that can be achieved. Therefore, a fair way to compare two models predicting the same outcome is a direct comparison in the same external validation data set. Unfortunately, the authors only compared the performance of the UK‐PSC score with the Mayo score and aspartate aminotransferase‐to‐platelet ratio index and did not assess the Amsterdam‐Oxford PSC score as well. For an indirect comparison of the UK‐PSC score with the Amsterdam‐Oxford score, we would like to raise the following points: The short‐term 2‐year risk score (RS ST ) takes advantage of the fact that early events are easier to predict. If we apply the Amsterdam‐Oxford score only for the first 2 years, the C statistic is 0.76 (95% confidence interval [CI] 0.58‐0.94) in our external validation instead of 0.68. For the 10‐year risk of outcome score (RS If we evaluate the Amsterdam‐Oxford score only for events within 10 years, the C statistic is 0.71 (95% CI 0.62‐0.80) in our external validation. LT ), the longest prediction is 10 years (prediction at 2 years after diagnosis over the 8 following years), whereas the Amsterdam‐Oxford PSC score was evaluated with its prediction up to 30 years from time of diagnosis. From this figure, one cannot conclude that the calibration of RS Furthermore, in Figure 3, calibration is not assessed by comparing predicted survival versus observed survival probability: it is actually between observed survival in the derivation cohort versus the validation cohort. LT is good. From a practical point of view, RS LT needs information both at T0 and T2; thus, prediction is only possible at T2. This is undesirable when counselling patients as to their prognosis after establishing a diagnosis of PSC. Instead of building a model at 2 years, an alternative method is to apply a so‐called dynamic prediction model, which can better reflect the dynamics of the disease progress than a baseline model. Copyright: Copyright 2020 Elsevier B.V., All rights reserved.

PY - 2020/1/1

Y1 - 2020/1/1

UR - http://www.scopus.com/inward/record.url?scp=85076878214&partnerID=8YFLogxK

U2 - https://doi.org/10.1002/hep.30964

DO - https://doi.org/10.1002/hep.30964

M3 - Letter

C2 - 31544247

SN - 0270-9139

VL - 71

SP - 398

EP - 399

JO - Hepatology (Baltimore, Md.)

JF - Hepatology (Baltimore, Md.)

IS - 1

ER -