TY - JOUR
T1 - Almost the best of three worlds
T2 - Risk, consistency and optional stopping for the switch criterion in nested model selection
AU - Van Der Pas, Stéphanie
AU - Grünwald, Peter
N1 - Funding Information: Our central result appeared in van der Pas (2013) for the special case m1 = 1 and m0 = 0, but the proof there contained an error. We are grateful to Tim van Erven for pointing this out to us. We are also thankful to the anonymous referees and to Hannes Leeb for raising the issue of whether the switch distribution has a 'special' place on the spectrum of a model selection criterion's possible risk and consistency behaviors. This research was supported by NWO VICI Project 639.073.04. Funding Information: Our central result appeared in van der Pas (2013) for the special case m1 = 1 and m0 = 0, but the proof there contained an error. We are grateful to Tim van Erven for pointing this out to us. We are also thankful to the anonymous referees and to Hannes Leeb for raising the issue of whether the switch distribution has a ‘special’ place on the spectrum of a model selection criterion’s possible risk and consistency behaviors. This research was supported by NWO VICI Project 639.073.04. Copyright: Copyright 2018 Elsevier B.V., All rights reserved.
PY - 2018/1
Y1 - 2018/1
N2 - We study the switch distribution, introduced by van Erven, Grünwald and De Rooij (2012), applied to model selection and subsequent estimation. While switching was known to be strongly consistent, here we show that it achieves minimax optimal parametric risk rates up to a log log n factor when comparing two nested exponential families, partially confirming a conjecture by Lauritzen (2012) and Cavanaugh (2012) that switching behaves asymptotically like the HannanQuinn criterion. Moreover, like Bayes factor model selection, but unlike standard significance testing, when one of the models represents a simple hypothesis, the switch criterion defines a robust null hypothesis test, meaning that its Type-I error probability can be bounded irrespective of the stopping rule. Hence, switching is consistent, insensitive to optional stopping and almost minimax risk optimal, showing that, Yang's (2005) impossibility result notwithstanding, it is possible to 'almost' combine the strengths of AIC and Bayes factor model selection.
AB - We study the switch distribution, introduced by van Erven, Grünwald and De Rooij (2012), applied to model selection and subsequent estimation. While switching was known to be strongly consistent, here we show that it achieves minimax optimal parametric risk rates up to a log log n factor when comparing two nested exponential families, partially confirming a conjecture by Lauritzen (2012) and Cavanaugh (2012) that switching behaves asymptotically like the HannanQuinn criterion. Moreover, like Bayes factor model selection, but unlike standard significance testing, when one of the models represents a simple hypothesis, the switch criterion defines a robust null hypothesis test, meaning that its Type-I error probability can be bounded irrespective of the stopping rule. Hence, switching is consistent, insensitive to optional stopping and almost minimax risk optimal, showing that, Yang's (2005) impossibility result notwithstanding, it is possible to 'almost' combine the strengths of AIC and Bayes factor model selection.
KW - AIC-BIC dilemma
KW - Consistency
KW - Exponential family
KW - Model selection
KW - Optional stopping
KW - Post model selection estimation
KW - Switch distribution
KW - Worst-case risk
UR - http://www.scopus.com/inward/record.url?scp=85040104330&partnerID=8YFLogxK
U2 - https://doi.org/10.5705/ss.202016.0011
DO - https://doi.org/10.5705/ss.202016.0011
M3 - Article
SN - 1017-0405
VL - 28
SP - 229
EP - 253
JO - Statistica Sinica
JF - Statistica Sinica
IS - 1
ER -