Please use this identifier to cite or link to this item: https://hdl.handle.net/20.500.11851/8615
Title: Understanding and Predicting Characteristics of Test Collections in Information Retrieval
Authors: Rahman M.M.
Kutlu, Mücahid
Lease M.
Keywords: Evaluation
Pooling
Reusability
Test collections
Information retrieval
Testing
Community evaluations
Document ranking
Evaluation
Human assessors
Participating teams
Pooling
Relevance judgement
Research communities
Test Collection
Text retrieval conferences
Reusability
Issue Date: 2022
Publisher: Springer Science and Business Media Deutschland GmbH
Source: Rahman, M., Kutlu, M., & Lease, M. (2022, February). Understanding and Predicting Characteristics of Test Collections in Information Retrieval. In International Conference on Information (pp. 136-148). Springer, Cham.
Abstract: Research community evaluations in information retrieval, such as NIST’s Text REtrieval Conference (TREC), build reusable test collections by pooling document rankings submitted by many teams. Naturally, the quality of the resulting test collection thus greatly depends on the number of participating teams and the quality of their submitted runs. In this work, we investigate: i) how the number of participants, coupled with other factors, affects the quality of a test collection; and ii) whether the quality of a test collection can be inferred prior to collecting relevance judgments from human assessors. Experiments conducted on six TREC collections illustrate how the number of teams interacts with various other factors to influence the resulting quality of test collections. We also show that the reusability of a test collection can be predicted with high accuracy when the same document collection is used for successive years in an evaluation campaign, as is common in TREC. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.
Description: 17th International Conference on Information for a Better World: Shaping the Global Future, iConference 2022 -- 28 February 2022 through 4 March 2022 -- -- 273989
URI: https://doi.org/10.1007/978-3-030-96960-8_10
https://hdl.handle.net/20.500.11851/8615
ISBN: 9783030969592
ISSN: 03029743
Appears in Collections:Bilgisayar Mühendisliği Bölümü / Department of Computer Engineering
Scopus İndeksli Yayınlar Koleksiyonu / Scopus Indexed Publications Collection

Show full item record

CORE Recommender

Page view(s)

26
checked on Dec 26, 2022

Google ScholarTM

Check

Altmetric


Items in GCRIS Repository are protected by copyright, with all rights reserved, unless otherwise indicated.