Comments for Probably Irrelevant

Comment on Query logs and information retrieval research by Lee S Jensen

Lee S Jensen — Tue, 22 Jun 2010 14:21:52 +0000

AOL got in big trouble when it released its search log. This was because it violated privacy issues. However, with a vertical search system the logs don’t generally have that concern. Being able to track the search path of person when they are only looking at genealogy (www.ancestry.com) or cameras (www.dpreview.com) doesn’t really tell you anything valuable about a particular person. Focusing on getting search logs from these types of companies should be our focus.

Comment on Query logs and information retrieval research by neelannair

neelannair — Wed, 09 Jun 2010 14:57:38 +0000

Valid points.

But I guess query logs are a huge investment for most search engines. It forms part of their competitive advantage over other search engines.

I’d like to propose a time-out clause for search engine query logs. They could release query logs more than ‘n’ years old for a license and fee. So Universities can give open access to researchers to work with them.

As far as the Lemur Project was concerned, I came to know about it in the news that it had been discontinued. There is such a thing as publicity, I guess. Wonder if they tied up with existing search engines or something to promote the cause.

Comment on Vintage Cornell/SMART Tech Reports? by chris

chris — Wed, 05 May 2010 21:58:00 +0000

This?

http://www.sigir.org/museum/

Comment on micro-IR by Gene Golovchinsky

Gene Golovchinsky — Mon, 14 Sep 2009 18:07:45 +0000

I think one of the key issues here is whether the retrieved information is actionable, that is, whether enough context is represented in the system to suggest meaningful actions based on retrieved results. Here's my take.

Comment on micro-IR by Daniel Tunkelang

Daniel Tunkelang — Sat, 12 Sep 2009 18:39:38 +0000

Great post! I just posted my own reaction here:

http://thenoisychannel.com/2009/09/12/micro-vs-macro-information-retrieval/

Comment on micro-IR by Jinyoung

Jinyoung — Sat, 12 Sep 2009 03:52:17 +0000

Hi, Miles. (we met in ECIR)

I agree with Fernando in that we’ve been dealing with very constrained features (i.e. textual similarity) in traditional IR.

What characterize Micro IR seems to be that the context (searcher goal) is known, with domain-specific notion of relevance (goodness) and similarity measures. I guess current IR frameworks (including learning to rank) can accmmodiate most of these problems. After all, it’s still about combining evidences, albeit of a different type.

This also reminds me of vertical search, where we need to infer user’s information goal given only query-words. Here, each information type may have somewhat different notion of relevance as well, although not to the extent you talk about in Micro IR.

Comment on micro-IR by Fernando

Fernando — Sat, 12 Sep 2009 00:57:22 +0000

Any particular information retrieval problem is a function of queries and documents. I try to approach this in as general a way as possible. For me, the only difference between micro and macro IR is text. In fact, I would say, in many ways, pure text IR is _more_ constrained than other IR tasks you mention.

Comment on Finding relevance judgements in the wild by mariana_soffer

mariana_soffer — Mon, 25 May 2009 14:11:34 +0000

And you complain, came on. Imagine if you have to do the stuff in spanish (not to mention other languages that are even more wierd), how do you parse? where do you get your training sets from? not even webscraping works here.

Comment on Finding relevance judgements in the wild by Jon

Jon — Mon, 20 Apr 2009 14:36:56 +0000

I think your intuition is correct about the FIRST method. First-posts also tend to be more verbose than answers — users often respond with a single sentence, which tends to be generally unhelpful in ranking.

Due to time constraints & such, we weren’t able to annotate many more threads for relevance. We identified 17k “candidate” question-post/answer-link pairs in the collection, using a few simple heuristics such as the presence of a link in a response message. Of those 17k, we annotated 550 as to whether or not they actually contained a question/answer pair and identified the 48 we used in the study. So, we found that roughly 8% of those candidates contain a real question/answer pair, and extrapolating up to the full 17k, I would estimate there are about 1400 question-answer pairs in the collection — that’s quite a few still to be found. I’m sure we didn’t find them all.

Ideally, of course, we’d like to use more queries. I don’t know at this point whether we’ll push forward with this type of test set creation, or whether we’ll do a more traditional relevance assessment. I’m tempted to do the latter, particularly because it would be nice to see if we observe the same results with the different types of test collections.

Comment on Finding relevance judgements in the wild by William Webber

William Webber — Mon, 20 Apr 2009 10:10:01 +0000

OK, I’ve properly read (rather than skimmed) your poster now. Very interesting work!

With regard to the relatively good performance of the FIRST method,
is this because the opening query in the linked-from thread is often similar to the opening query in the linked-to thread?

It is rather disappointing that you were only able to identify 48 query/answer pairs out of 375,000 threads. Is this because that
was all there was, or did you stop looking once you’d found 48?