Good to read Sakai's reply to Fuhr’s Guidelines for Information Retrieval Evaluation:
@djoerd Sakai made some good points. But. MAP's user model was reverse engineered decades after its inception. And what user model would actually consider the differences between ranks 1 and 2 and between 2 and inf. the same?
Maybe we as an IR community should identify classes of user models (e.g. "adhoc", "automated") and identify the best known measure for each class. Otherwise papers will be tempted to cherry-pick the measure that best shows the advantage of the paper's contribution.
@arjen @djoerd anything@20 seems like a strange result page measure since that eyetracking paper by Joachims showed most web users can't get themselves to scroll to the 7th result item. But it's good to know there are standard measures. Maybe PCs can adopt a good list of dos and don'ts incorporating the widely accepted ones of the Fuhr criteria and things like these standard measures.
The "unofficial" Information Retrieval Mastodon Instance.
Goal: Make idf.social a viable and valuable social space for anyone working in Information Retrieval and related scientific research.
Everyone welcome but expect some level of geekiness on the instance and federated timelines.