Oops MRR, triggering a strong Norbert Fuhr gets angry feeling.

@arjen but... R-precision = MRR if there's just one relevant result. Angry about precision too?

@djoerd Norbert Fuhr complained about MRR as used in a critical article for the SIGIR Forum, paragraph 2.1: dl.acm.org/citation.cfm?id=319

I think in this specific case, the problem did not assume 1 single answer.

I think that, indeed, averaging the Rprec is problematic too for same reasons, but might be justified probabilistic ally? Not sure...

@arjen Fuhr's concerns are interesting and relevant, but if my user model assumes: "go down the rank list until you find the first relevant document, then stop" (which is not unrealistic) then what should I measure?

@djoerd you could accumulate number of docs accessed (and minimize that) instead of doing the reciprocal, just as one example! ESL would be interesting.

@arjen good suggestions!

The "unofficial" Information Retrieval Mastodon Instance.

Goal: Make idf.social a viable and valuable social space for anyone working in Information Retrieval and related scientific research.

Everyone welcome but expect some level of geekiness on the instance and federated timelines.