The AMSTAR-2 critical appraisal tool and editorial decision-making for systematic reviews: Retrospective, bibliometric study

AMSTAR-2 is a critical appraisal instrument for systematic reviews and may have a role in editorial processes. This study explored whether associations exist between AMSTAR-2 assessments and editorial decisions. A retrospective, cross-sectional study of manuscripts submitted to a single journal between 2015 and 2017 was undertaken. All submissions that reported an eligible systematic review were assessed using AMSTAR-2 by two assessors. Inter-rater agreement (IRR) was calculated for all AMSTAR-2 items. Associations between AMSTAR-2 assessments and the editorial decision, final publication status in any journal, and measures of impact were explored. One hundred and twenty-two manuscripts were included. Across all AMSTAR-2 items, the IRR varied from 0.03 (slight agreement) to 0.82 (substantial agreement). All submissions contained at least two critical methodological weaknesses. There was no difference in the number of weaknesses (median: 4; IQR: 3–5 vs. median: 4; IQR: 3.5–4.5; p = 0.482) between accepted and rejected submissions. Neither was there a difference between rejected submissions published elsewhere and those which remained unpublished (median: 4; IQR: 3.5–4.5 vs. median: 4; IQR: 4.5–5; p = 0.103). The number of weaknesses was not associated with academic impact. There was no association with AMSTAR-2 assessments and editorial outcomes. Further work is required to explore whether the instrument can be prospectively operationalized for use during editorial processes.