Oops MRR, triggering a strong Norbert Fuhr gets angry feeling.
@arjen but... R-precision = MRR if there's just one relevant result. Angry about precision too?
I think in this specific case, the problem did not assume 1 single answer.
I think that, indeed, averaging the Rprec is problematic too for same reasons, but might be justified probabilistic ally? Not sure...
@arjen Fuhr's concerns are interesting and relevant, but if my user model assumes: "go down the rank list until you find the first relevant document, then stop" (which is not unrealistic) then what should I measure?
@djoerd you could accumulate number of docs accessed (and minimize that) instead of doing the reciprocal, just as one example! ESL would be interesting.
@arjen good suggestions!
The "unofficial" Information Retrieval Mastodon Instance.
Goal: Make idf.social a viable and valuable social space for anyone working in Information Retrieval and related scientific research.
Everyone welcome but expect some level of geekiness on the instance and federated timelines.