TY - JOUR
T1 - Accumulation bias in meta-analysis
T2 - The need to consider time in error control [version 1; peer review: 2 approved]
AU - Ter Schure, Judith
AU - Grünwald, Peter
N1 - Funding Information: This work is part of the NWO TOP-I research programme Safe Bayesian Inference [617.001.651], which is financed by the Netherlands Organisation for Scientific Research (NWO). Publisher Copyright: © 2019 ter Schure J and Grünwald P.
PY - 2019
Y1 - 2019
N2 - Studies accumulate over time and meta-analyses are mainly retrospective. These two characteristics introduce dependencies between the analysis time, at which a series of studies is up for meta-analysis, and results within the series. Dependencies introduce bias - Accumulation Bias - and invalidate the sampling distribution assumed for p-value tests, thus inflating type-I errors. But dependencies are also inevitable, since for science to accumulate efficiently, new research needs to be informed by past results. Here, we investigate various ways in which time influences error control in meta-analysis testing. We introduce an Accumulation Bias Framework that allows us to model a wide variety of practically occurring dependencies including study series accumulation, meta-analysis timing, and approaches to multiple testing in living systematic reviews. The strength of this framework is that it shows how all dependencies affect p-value-based tests in a similar manner. This leads to two main conclusions. First, Accumulation Bias is inevitable, and even if it can be approximated and accounted for, no valid p-value tests can be constructed. Second, tests based on likelihood ratios withstand Accumulation Bias: they provide bounds on error probabilities that remain valid despite the bias. We leave the reader with a choice between two proposals to consider time in error control: either treat individual (primary) studies and meta-analyses as two separate worlds - each with their own timing - or integrate individual studies in the meta-analysis world. Taking up likelihood ratios in either approach allows for valid tests that relate well to the accumulating nature of scientific knowledge. Likelihood ratios can be interpreted as betting profits, earned in previous studies and invested in new ones, while the meta-analyst is allowed to cash out at any time and advice against future studies.
AB - Studies accumulate over time and meta-analyses are mainly retrospective. These two characteristics introduce dependencies between the analysis time, at which a series of studies is up for meta-analysis, and results within the series. Dependencies introduce bias - Accumulation Bias - and invalidate the sampling distribution assumed for p-value tests, thus inflating type-I errors. But dependencies are also inevitable, since for science to accumulate efficiently, new research needs to be informed by past results. Here, we investigate various ways in which time influences error control in meta-analysis testing. We introduce an Accumulation Bias Framework that allows us to model a wide variety of practically occurring dependencies including study series accumulation, meta-analysis timing, and approaches to multiple testing in living systematic reviews. The strength of this framework is that it shows how all dependencies affect p-value-based tests in a similar manner. This leads to two main conclusions. First, Accumulation Bias is inevitable, and even if it can be approximated and accounted for, no valid p-value tests can be constructed. Second, tests based on likelihood ratios withstand Accumulation Bias: they provide bounds on error probabilities that remain valid despite the bias. We leave the reader with a choice between two proposals to consider time in error control: either treat individual (primary) studies and meta-analyses as two separate worlds - each with their own timing - or integrate individual studies in the meta-analysis world. Taking up likelihood ratios in either approach allows for valid tests that relate well to the accumulating nature of scientific knowledge. Likelihood ratios can be interpreted as betting profits, earned in previous studies and invested in new ones, while the meta-analyst is allowed to cash out at any time and advice against future studies.
KW - Accumulation bias
KW - Cumulative
KW - Evidence-based research
KW - Likelihood ratio
KW - Living systematic reviews
KW - Meta-analysis
KW - Research waste
KW - Sequential
UR - http://www.scopus.com/inward/record.url?scp=85075195108&partnerID=8YFLogxK
U2 - https://doi.org/10.12688/F1000RESEARCH.19375.1
DO - https://doi.org/10.12688/F1000RESEARCH.19375.1
M3 - Article
C2 - 31737258
SN - 2046-1402
VL - 8
JO - F1000Research
JF - F1000Research
M1 - 962
ER -