http://thenoisychannel.com/2009/09/12/micro-vs-macro-information-retrieval/

I agree with Fernando in that we’ve been dealing with very constrained features (i.e. textual similarity) in traditional IR.

What characterize Micro IR seems to be that the context (searcher goal) is known, with domain-specific notion of relevance (goodness) and similarity measures. I guess current IR frameworks (including learning to rank) can accmmodiate most of these problems. After all, it’s still about combining evidences, albeit of a different type.

This also reminds me of vertical search, where we need to infer user’s information goal given only query-words. Here, each information type may have somewhat different notion of relevance as well, although not to the extent you talk about in Micro IR.

Due to time constraints & such, we weren’t able to annotate many more threads for relevance. We identified 17k “candidate” question-post/answer-link pairs in the collection, using a few simple heuristics such as the presence of a link in a response message. Of those 17k, we annotated 550 as to whether or not they actually contained a question/answer pair and identified the 48 we used in the study. So, we found that roughly 8% of those candidates contain a real question/answer pair, and extrapolating up to the full 17k, I would estimate there are about 1400 question-answer pairs in the collection — that’s quite a few still to be found. I’m sure we didn’t find them all.

Ideally, of course, we’d like to use more queries. I don’t know at this point whether we’ll push forward with this type of test set creation, or whether we’ll do a more traditional relevance assessment. I’m tempted to do the latter, particularly because it would be nice to see if we observe the same results with the different types of test collections.

With regard to the relatively good performance of the FIRST method,
is this because the opening query in the linked-from thread is often similar to the opening query in the linked-to thread?

It is rather disappointing that you were only able to identify 48 query/answer pairs out of 375,000 threads. Is this because that
was all there was, or did you stop looking once you’d found 48?

In all three cases I linked to in the post (and most of the cases we used in our evaluation) the original user who asked the question confirms that the linked-to answer thread is in fact relevant to their question in a subsequent post. If the original poster indicates that the linked-to thread is not relevant, then clearly that shouldn’t be used in an evaluation. If there’s no indication at all, then we’re back to the same situation that we see in TREC — where the annotators (or “linkers”) are different than the user who issues the query.

There’s a few aspects to the evaluation: the query, which is representative of some information need, and the document assessment. Ideally, we would like:
(1) to get a query or other description of the information need and the relevance assessment from the same person, to ensure there is agreement between the two
(2) that person to be actually performing the task in question, yielding queries & information needs we know to be representative of the task
(3) (nearly) exhaustive relevance assessment.
But, this is never the case in practice.

Sometimes we have a query log, which provides highly representative queries. In those cases the assessment is done someone who didn’t originally provide the query and the information need description is “back fit” to the query observed in the log. In this case, we cannot ensure criteria (1)

Sometimes we don’t have a query log, in which case the participants or assessors make an attempt to develop information needs that they believe are representative of the task, and then those same people assess those documents for relevance. In this case we cannot ensure criteria (2)

In the case I described in the blog post, we do meet criteria (2), and maybe (1), but we cannot ensure criteria (3).