Comments on: Blogs, queries, corpora

By: diazf

diazf — Fri, 12 Sep 2008 16:10:35 +0000

Hey Craig. Thank you for clarifying the process of developing the Blog tracks. We’ll have to discuss getting you those query logs over a pint sometime.

By: Craig Macdonald

Craig Macdonald — Fri, 12 Sep 2008 15:17:48 +0000

While I may be partial to an occasional pint of beer, I would have to say that a great deal of thought goes into the defining of a TREC track and the corresponding tasks! Tasks must be proposed and motivated prior to TREC, in proposals which are rated by the TREC program committee. In essence, we didn't come up with them on the back of a beer mat. You suggest a query log analysis to motivate the tasks. Why indeed, what a great idea! Indeed, its funny that the opening of each TREC Blog track overview describes the tasks, and motivates them using a study of blog search query logs. Actually, if you care to read the overview papers or the ICSWM paper, you'll find that all opinion finding queries for TREC 06 and 07 where sampled from a real query log. Nevertheless Fernando, if your company is able to provide more up-to-date query logs to allow the tasks to be refined, your help would be much appreciated. Further reading: 1. Overview of TREC Blog track 2007. C. Macdonald, I. Ounis & I. Soboroff. TREC 2007 Proceedings. 2. Overview of TREC Blog track 2006. I. Ounis et al. TREC 2006 Proceedings. 3. On TREC Blog Track. I. Ounis et al. Proceedings of ICSWM 2008. [Video]. 4. A Study of Blog Search. G. Mishne & M. de Rijke. Proceedings of ECIR 2006.

By: Jon Elsas

Jon Elsas — Fri, 12 Sep 2008 05:39:52 +0000

Fernando -- Although blog search is an area that I have had a lot of fun working in, I agree that the tasks need to be better defined and more rooted in reality. This really became evident in last year's TREC track. Like other nascent trec tracks, the participants developed queries, and then judged them after runs were submitted. Some of the submitted queries really seemed like oddball blog search queries ("christmas") and the corresponding relevance judgements were unrealistic for a real web search task (> 100 relevant blogs). A tighter task definition and queries from real query logs would've certainly helped the situation. But, I don't think the TREC blog tasks are as separated from reality as you seem to imply. No doubt the research community can benefit from better understanding of queries and information needs -- and that goes for pretty much every IR task considered at TREC. I think, at least to some degree, the blog track tasks were inspired by some commercial examples of blog information access. The distillation task is really doing what google is doing at the top of their blog search list. Bloglines is another good example. The opinion task is similar to sentiment mining services BlogPulse provided for corporate customers (based on my understanding from previous discussions with Matt Hurst & Natalie Glance).