Comments on: What is different about highly effective retrievals?

By: miles

miles — Sun, 21 Sep 2008 20:01:21 +0000

Fernando– I suspect you’re right, and it is likely that the top-performing systems are retrieving unjudged relevant documents. But by virtue of _being_ the best performers, they are also retrieving the judged relevant docs. So it’s easy for me to believe that there are some very good systems far down in the rankings (having had the bad luck to retrieve unjudged relevant docs). But it’s not clear to me how relevant documents that weren’t judged due to pooling would hurt the relret for high performers.

By: Fernando

Fernando — Sun, 21 Sep 2008 15:48:54 +0000

A few explanations for why your top runs may have low relret

pooling depth. compute metric for the pooling depth. there could be unjudged relevant documents. automatic runs may be saved because their unjudged documents are more likely to be within in the pooling depth of another run.
manual runs may be high precision, as opposed to high recall and we can certainly have methods with high precision which are low recall.

By: Fernando

Fernando — Sun, 21 Sep 2008 15:48:28 +0000

Before starting a discussion about performance prediction the following questions need to be answered,

why are you interested in performance prediction? do you want to merge results? abandon (or select) a ranking system for the query? abandon (or select) a ranking a system system for all queries? not all of these suggest the optimizing the same metric.
do you really not have relevance information? performance predictors often provide relevance surrogates (e.g. frequently retrieved documents across systems). does your system—from cold-start or warmed-up—have other sources of relevance surrogates?

By: Jon Elsas

Jon Elsas — Sat, 20 Sep 2008 04:27:35 +0000

Brendan — thanks for the links — great posts.

Judgement-free evaluation has utility beyond absolute system assessment. In meta search, or unsupervised rank aggregation, we often want to know which system’s output is more reliable at query time in order to inform how results are merged.

By: Brendan O'Connor

Brendan O'Connor — Fri, 19 Sep 2008 22:46:33 +0000

Why not always use human-judged relevance information? On Amazon Mech. Turk you can get hundreds of relevance judgments for a dollar.

Sorry to promote my own posts, but here’s the evidence…

http://blog.doloreslabs.com/2008/09/amt-fast-cheap-good-machine-learning/
http://blog.doloreslabs.com/2008/04/search-engine-relevance-an-empirical-test/