TY - GEN
T1 - CLEF 2019 technology assisted reviews in empirical medicine overview
AU - Kanoulas, E.
AU - Li, D.
AU - Azzopardi, L.
AU - Spijker, R.
PY - 2019
Y1 - 2019
N2 - Systematic reviews are a widely used method to provide an overview over the current scientific consensus, by bringing together multiple studies in a systematic, reliable, and transparent way. The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying all relevant studies in an unbiased way both complex and time consuming to the extent that jeopardizes the validity of their findings and the ability to inform policy and practice in a timely manner. The CLEF 2019 e-Health TAR Lab accommodated two tasks. Task 1 focused on retrieving relevant studies from PubMed without the use of a Boolean query, while Task 2 focused on the efficient and effective ranking of studies during the abstract and title screening phase of conducting a systematic review. In the 2019 lab we also expanded upon the type of systematics reviews considered. Hence, beyond Diagnostic Test Accuracy reviews, we also included Intervention, Prognosis, and Qualitative systematic reviews. We constructed a benchmark collection of 31 reviews published by Cochrane, and the corresponding relevant and irrelevant articles found by the original Boolean query. Three teams participated in Task 2, submitting automatic and semi-automatic runs, using information retrieval and machine learning algorithms over a variety of text representations, in a batch and iterative manner. This paper reports both the methodology used to construct the benchmark collection, and the results of the evaluation.
AB - Systematic reviews are a widely used method to provide an overview over the current scientific consensus, by bringing together multiple studies in a systematic, reliable, and transparent way. The large and growing number of published studies, and their increasing rate of publication, makes the task of identifying all relevant studies in an unbiased way both complex and time consuming to the extent that jeopardizes the validity of their findings and the ability to inform policy and practice in a timely manner. The CLEF 2019 e-Health TAR Lab accommodated two tasks. Task 1 focused on retrieving relevant studies from PubMed without the use of a Boolean query, while Task 2 focused on the efficient and effective ranking of studies during the abstract and title screening phase of conducting a systematic review. In the 2019 lab we also expanded upon the type of systematics reviews considered. Hence, beyond Diagnostic Test Accuracy reviews, we also included Intervention, Prognosis, and Qualitative systematic reviews. We constructed a benchmark collection of 31 reviews published by Cochrane, and the corresponding relevant and irrelevant articles found by the original Boolean query. Three teams participated in Task 2, submitting automatic and semi-automatic runs, using information retrieval and machine learning algorithms over a variety of text representations, in a batch and iterative manner. This paper reports both the methodology used to construct the benchmark collection, and the results of the evaluation.
KW - Active Learning
KW - Evaluation
KW - Information Retrieval
KW - Systematic Reviews
KW - TAR
KW - Text Classification
UR - http://ceur-ws.org/Vol-2380/
UR - http://www.scopus.com/inward/record.url?scp=85070491385&partnerID=8YFLogxK
UR - https://www.scopus.com/inward/record.uri?partnerID=HzOxMe3b&scp=85070491385&origin=inward
M3 - Conference contribution
VL - 2380
T3 - CEUR Workshop Proceedings
BT - Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum
A2 - Cappellato, L.
A2 - Ferro, N.
A2 - Losada, D.E.
A2 - Müller, H.
A2 - Ferro, Nicola
A2 - Losada, David E.
A2 - Muller, Henning
A2 - Cappellato, Linda
PB - CEUR-WS
CY - Aachen
T2 - 10th International Conference of the CLEF Association, CLEF 2019
Y2 - 9 September 2019 through 12 September 2019
ER -