TY - JOUR
T1 - A puzzle of proportions: Two popular Bayesian tests can yield dramatically different conclusions
AU - Dablander, Fabian
AU - Huth, K.B.S.
AU - Gronau, Quentin
AU - Etz, Alexander
AU - Wagenmakers, Eric-Jan
PY - 2022/4/15
Y1 - 2022/4/15
N2 - Testing the equality of two proportions is a common procedure in science, especially in medicine and public health. In these domains, it is crucial to be able to quantify evidence for the absence of a treatment effect. Bayesian hypothesis testing by means of the Bayes factor provides one avenue to do so, requiring the specification of prior distributions for parameters. The most popular analysis approach views the comparison of proportions from a contingency table perspective, assigning prior distributions directly to the two proportions. Another, less popular approach views the problem from a logistic regression perspective, assigning prior distributions to logit-transformed parameters. Reanalyzing 39 null results from the New England Journal of Medicine with both approaches, we find that they can lead to markedly different conclusions, especially when the observed proportions are at the extremes (ie, very low or very high). We explain these stark differences and provide recommendations for researchers interested in testing the equality of two proportions and users of Bayes factors more generally. The test that assigns prior distributions to logit-transformed parameters creates prior dependence between the two proportions and yields weaker evidence when the observations are at the extremes. When comparing two proportions, we argue that this test should become the new default.
AB - Testing the equality of two proportions is a common procedure in science, especially in medicine and public health. In these domains, it is crucial to be able to quantify evidence for the absence of a treatment effect. Bayesian hypothesis testing by means of the Bayes factor provides one avenue to do so, requiring the specification of prior distributions for parameters. The most popular analysis approach views the comparison of proportions from a contingency table perspective, assigning prior distributions directly to the two proportions. Another, less popular approach views the problem from a logistic regression perspective, assigning prior distributions to logit-transformed parameters. Reanalyzing 39 null results from the New England Journal of Medicine with both approaches, we find that they can lead to markedly different conclusions, especially when the observed proportions are at the extremes (ie, very low or very high). We explain these stark differences and provide recommendations for researchers interested in testing the equality of two proportions and users of Bayes factors more generally. The test that assigns prior distributions to logit-transformed parameters creates prior dependence between the two proportions and yields weaker evidence when the observations are at the extremes. When comparing two proportions, we argue that this test should become the new default.
KW - Bayesian testing
KW - data reanalysis
KW - equality of proportions
KW - evidence of absence
UR - http://www.scopus.com/inward/record.url?scp=85120863751&partnerID=8YFLogxK
UR - https://github.com/fdabl/Proportion-Puzzle
U2 - https://doi.org/10.1002/sim.9278
DO - https://doi.org/10.1002/sim.9278
M3 - Article
C2 - 34897784
SN - 0277-6715
VL - 41
SP - 1319
EP - 1333
JO - Statistics in medicine
JF - Statistics in medicine
IS - 8
ER -