On the effect of relevance scales in crowdsourcing relevance assessments for Information Retrieval evaluation

Volume: 58, Issue: 6, Pages: 102688 - 102688
Published: Nov 1, 2021
Abstract
Relevance is a key concept in information retrieval and widely used for the evaluation of search systems using test collections. We present a comprehensive study of the effect of the choice of relevance scales on the evaluation of information retrieval systems. Our work analyzes and compares four crowdsourced scales (2-levels, 4-levels, and 100-levels ordinal scales, and a magnitude estimation scale) and two expert-labeled datasets (on 2- and...
Paper Details
Title
On the effect of relevance scales in crowdsourcing relevance assessments for Information Retrieval evaluation
Published Date
Nov 1, 2021
Volume
58
Issue
6
Pages
102688 - 102688
Citation AnalysisPro
  • Scinapse’s Top 10 Citation Journals & Affiliations graph reveals the quality and authenticity of citations received by a paper.
  • Discover whether citations have been inflated due to self-citations, or if citations include institutional bias.