Finding relevance judgements in the wild
We recently heard our poster on online forum search was accepted to SIGIR 09, and I’ve been wanting to post something about the test setup we used in that study.
There’s no existing IR test collection for such a task, although some similar datasets do exist. For various reasons we weren’t able to create a traditional test collection, with user-issued queries and deep pools of relevance judgements. But, this particular dataset and possibly other online dialog archives can be mined to produce a ready-made IR test collection.
The users of the online forum we’ve been looking at frequently include links in their forum posts — often to previous messages and threads in the same forum. These links are sometimes in response to a new user’s question, and refer the user to a previous instance of the same (or similar) question and an answer contributed by another user. Here’s a few examples to illustrate my point. This interaction among forum users can be used as a form of query/relevance judgement pair. See the paper for a few more details on how we characterize the presence of a question-post/answer-link pair.
This type of test collection creation does have some distinct advantages over the typical retrieval test collections used at TREC. First, the queries represent real information needs of real users of the online forum. Many TREC queries are pulled from search engine logs, but frequently (as in the Blog Track’s Feed Distillation task) the queries are invented by participants or assessors. The information needs present in the online forum posts are much more verbose than typical keyword queries on a web search engine, providing a retrieval system more evidence with which to use in relevance scoring. The “relevance judgement”, provided by another forum user linking to a previous thread, also presents in-situ relevance information — sensitive not only to the original question, but also to the overall nature of the forum and the time when the question was asked.
There are several drawbacks inherent in this type of corpus creation, most importantly with regard to the exhaustiveness of the relevance assessment. Typically in TREC-style collection development, ranked results from several retrieval systems are pooled and those pooled documents are assessed for relevance. When the systems’ output is sufficiently diverse and relevance assessment is sufficiently deep, this produces a reasonably complete relevance assessment for each query — if a relevant document is in the collection, it would most likely be retrieved by one of the systems and be judged by being admitted into the pool. The method of collecting relevance judgements we use in our SIGIR poster, on the other hand, will not produce anything close to an exhaustive set of relevant threads. In the great majority of cases, only a single thread is linked to in a subsequent reply message. There is no guarantee that this thread is the best or only relevant thread in the collection. For this reason, we must take care not to assume non-judged threads are necessarily irrelevant.
There are plenty of datasets that seem to be ready-made for classification or regression tasks, without any need for annotation — for example the classic 20 newsgroups for text classification and Yahoo! Answers for a number of prediction tasks. For relevance ranking, however, I haven’t seen any ready-made datasets with real relevance judgements, as opposed to noisy interaction indicators such as click-through statistics. Conversation archives like the one we use offer one way to mine behavioral data for relevance judgements, offering ground-truth preferable in many ways to post-hoc relevance assessment.
Jon,
{{{
but frequently (as in the Blog Track) the queries are invented by participants or assessors.
}}}
This is at least inaccurate, if not misleading. We have repeatedly said that the queries used for the opinion finding and polarity tasks are driven from a real commercial query logs (including on this same blog). They are certainly not invented by participants (please read the TREC Blog track overview papers).
Only the topics used for the blog distillation task were made by the participating groups. However, they were certainly not invented (sic) by them. They are based on *rea*l information needs.
Iadh — Sorry for the inaccuracy in my post — I *meant* to refer to just the distillation task, not the queries used across the track. I’ve corrected the post to reflect this.
I agree with you that the queries reflect real information needs, but they were not harvested from query logs of a blog search engine. They were created by IR researchers trying to test their systems. I never actually went to a blog search engine and entered the query [celebrity babies], nor did I observe someone doing that. This was a query that I thought may be reflective of the types of queries such a system would receive. Do I *know* that the queries I generated are in fact representative of blog search queries? No, I don’t. I have no data to base my characterization of feed distillation queries on.
You might, and if you do, I’d love to hear your thoughts on how representative the feed distillation queries we generated are.
The query “celebrity babies” doesn’t seem odd at all.
http://www.google.com/insights/search/#q=%22celebrity%20babies%22&cmpt=q
The point is: there are tradeoffs in all evaluations. We would like to know whether system X performs better than system Y for a given task. But in practice, we always have to approximate that task somehow.
There’s a few aspects to the evaluation: the query, which is representative of some information need, and the document assessment. Ideally, we would like:
(1) to get a query or other description of the information need and the relevance assessment from the same person, to ensure there is agreement between the two
(2) that person to be actually performing the task in question, yielding queries & information needs we know to be representative of the task
(3) (nearly) exhaustive relevance assessment.
But, this is never the case in practice.
Sometimes we have a query log, which provides highly representative queries. In those cases the assessment is done someone who didn’t originally provide the query and the information need description is “back fit” to the query observed in the log. In this case, we cannot ensure criteria (1)
Sometimes we don’t have a query log, in which case the participants or assessors make an attempt to develop information needs that they believe are representative of the task, and then those same people assess those documents for relevance. In this case we cannot ensure criteria (2)
In the case I described in the blog post, we do meet criteria (2), and maybe (1), but we cannot ensure criteria (3).
A very nice idea, but as you say, the relationship between “linked” and relevant-by-annotation is problematic. Perhaps a useful experiment would be to have domain experts perform a standard relevance annotation run (without the prompting of links), and see what the overlap between linked and relevant was?
William – I agree it can be a little problematic. But, the nice thing about mining a conversation archive is that we can observe any feedback from the original asker.
In all three cases I linked to in the post (and most of the cases we used in our evaluation) the original user who asked the question confirms that the linked-to answer thread is in fact relevant to their question in a subsequent post. If the original poster indicates that the linked-to thread is not relevant, then clearly that shouldn’t be used in an evaluation. If there’s no indication at all, then we’re back to the same situation that we see in TREC — where the annotators (or “linkers”) are different than the user who issues the query.
OK, I’ve properly read (rather than skimmed) your poster now. Very interesting work!
With regard to the relatively good performance of the FIRST method,
is this because the opening query in the linked-from thread is often similar to the opening query in the linked-to thread?
It is rather disappointing that you were only able to identify 48 query/answer pairs out of 375,000 threads. Is this because that
was all there was, or did you stop looking once you’d found 48?
I think your intuition is correct about the FIRST method. First-posts also tend to be more verbose than answers — users often respond with a single sentence, which tends to be generally unhelpful in ranking.
Due to time constraints & such, we weren’t able to annotate many more threads for relevance. We identified 17k “candidate” question-post/answer-link pairs in the collection, using a few simple heuristics such as the presence of a link in a response message. Of those 17k, we annotated 550 as to whether or not they actually contained a question/answer pair and identified the 48 we used in the study. So, we found that roughly 8% of those candidates contain a real question/answer pair, and extrapolating up to the full 17k, I would estimate there are about 1400 question-answer pairs in the collection — that’s quite a few still to be found. I’m sure we didn’t find them all.
Ideally, of course, we’d like to use more queries. I don’t know at this point whether we’ll push forward with this type of test set creation, or whether we’ll do a more traditional relevance assessment. I’m tempted to do the latter, particularly because it would be nice to see if we observe the same results with the different types of test collections.
And you complain, came on. Imagine if you have to do the stuff in spanish (not to mention other languages that are even more wierd), how do you parse? where do you get your training sets from? not even webscraping works here.
I am surprised that out of a large sample of 17k you only selected 48 threads that were relevant to your study. Each test could of course produce different results next time with different criteria. So further experiments may give different results.