Norbert Fuhr's recommendations for gaining scientific knowledge from experiments:

1. Do not use MRR or MAP;
2. Instead of relative improvements, regard the effect size!
3. For multiple significance tests, use a correction, such as Bonferoni or Tukey's HSD (NB comparing only to the 2nd best method does not help!)
4. There are no significant improvements for re-usable test collections! (hypotheses have to be formulated before the work)

Sign in to participate in the conversation

The "unofficial" Information Retrieval Mastodon Instance.

Goal: Make a viable and valuable social space for anyone working in Information Retrieval and related scientific research.

Everyone welcome but expect some level of geekiness on the instance and federated timelines.