Please use this identifier to cite or link to this item:
Title: Constructing Test Collections using Multi-armed Bandits and Active Learning
Authors: Rahman, M. Mustafizur
Kutlu, Mücahid
Lease, Matthew
Keywords: Information retrieval 
 search engines 
relevance assessments
Issue Date: 2019
Publisher:  Association for Computing Machinery, Inc
Source: Rahman, M. M., Kutlu, M., and Lease, M. (2019, May). Constructing Test Collections using Multi-armed Bandits and Active Learning. In The World Wide Web Conference (pp. 3158-3164). ACM.
Abstract: While test collections provide the cornerstone of system-based evaluation in information retrieval, human relevance judging has become prohibitively expensive as collections have grown ever larger. Consequently, intelligently deciding which documents to judge has become increasingly important. We propose a two-phase approach to intelligent judging across topics which does not require document rankings from a shared task. In the first phase, we dynamically select the next topic to judge via a multi-armed bandit method. In the second phase, we employ active learning to select which document to judge next for that topic. Experiments on three TREC collections (varying scarcity of relevant documents) achieve ? ? 0.90 correlation for P@10 ranking and find 90% of the relevant documents at 48% of the original budget. To support reproducibility and follow-on work, we have shared our code online1. © 2019 IW3C2 (International World Wide Web Conference Committee), published under Creative Commons CC-BY 4.0 License.
Description: The Web Conference 2019 - Proceedings of the World Wide Web Conference (2019: San Francisco; United States )
ISBN: 9.78145E+12
Appears in Collections:Bilgisayar Mühendisliği Bölümü / Department of Computer Engineering
Scopus İndeksli Yayınlar Koleksiyonu / Scopus Indexed Publications Collection
WoS İndeksli Yayınlar Koleksiyonu / WoS Indexed Publications Collection

Show full item record

CORE Recommender


checked on Sep 23, 2022


checked on Sep 24, 2022

Page view(s)

checked on Dec 26, 2022

Google ScholarTM



Items in GCRIS Repository are protected by copyright, with all rights reserved, unless otherwise indicated.