Comments for Probably Irrelevant http://probablyirrelevant.org Information Retrieval Research and Development Thu, 30 Jun 2011 14:19:08 -0400 http://wordpress.org/?v=2.8.4 hourly 1 Comment on Informal SIGIR Test of Time Award by Ian http://probablyirrelevant.org/2011/06/informal-sigir-test-of-time-award/comment-page-1/#comment-16963 Ian Thu, 30 Jun 2011 14:19:08 +0000 http://probablyirrelevant.org/?p=112#comment-16963 Ahhh, 2001 was such an... eventful SIGIR Ahhh, 2001 was such an… eventful SIGIR

]]> Comment on IR in IBM’s Watson: An interview with Nico Schlaefer by Le Zhao http://probablyirrelevant.org/2011/03/ir-in-ibms-watson-an-interview-with-nico-schlaefer/comment-page-1/#comment-14229 Le Zhao Sat, 19 Mar 2011 21:50:46 +0000 http://probablyirrelevant.org/?p=100#comment-14229 Nice interview! The technique looks more like expanding an answer string? The use of topic modeling in assisting expansion looks interesting. Nice interview! The technique looks more like expanding an answer string? The use of topic modeling in assisting expansion looks interesting.

]]> Comment on Explicit negative feedback comes to the web…somewhat by miles http://probablyirrelevant.org/2011/02/explicit-negative-feedback-comes-to-the-web-somewhat/comment-page-1/#comment-13354 miles Wed, 16 Feb 2011 14:48:52 +0000 http://probablyirrelevant.org/?p=91#comment-13354 Fernando, my original comment stemmed from a (possibly wrong) assumption. But I think the assumption is plausible. Namely, I assumed that the information implied by an explicit block is qualitatively different from the implications of a generic click: The block is a strong statement of non-relevance, while a click entails a weak indication of relevance. So this type of negative feedback is indeed subject to click spam just like positive click-based feedback. But it seems to me that the spam detection methods should differ for negative feedback vs. positive feedback. Negative feedback isn't an intractable problem, just a different one than that posed by positive clickthrough feedback. Fernando, my original comment stemmed from a (possibly wrong) assumption. But I think the assumption is plausible. Namely, I assumed that the information implied by an explicit block is qualitatively different from the implications of a generic click: The block is a strong statement of non-relevance, while a click entails a weak indication of relevance.

So this type of negative feedback is indeed subject to click spam just like positive click-based feedback. But it seems to me that the spam detection methods should differ for negative feedback vs. positive feedback. Negative feedback isn’t an intractable problem, just a different one than that posed by positive clickthrough feedback.

]]> Comment on Explicit negative feedback comes to the web…somewhat by Jon http://probablyirrelevant.org/2011/02/explicit-negative-feedback-comes-to-the-web-somewhat/comment-page-1/#comment-13327 Jon Tue, 15 Feb 2011 18:32:27 +0000 http://probablyirrelevant.org/?p=91#comment-13327 In this case, at least, Google can (probably) tie the negative feedback to an individual. Just like a lot of crowd-sourced tasks, identifying quality is mostly a matter of weeding out the bad workers, finding the over-zealous reasonably-high-quality super-users, and focusing on them. In this case, at least, Google can (probably) tie the negative feedback to an individual. Just like a lot of crowd-sourced tasks, identifying quality is mostly a matter of weeding out the bad workers, finding the over-zealous reasonably-high-quality super-users, and focusing on them.

]]> Comment on Explicit negative feedback comes to the web…somewhat by Fernando http://probablyirrelevant.org/2011/02/explicit-negative-feedback-comes-to-the-web-somewhat/comment-page-1/#comment-13322 Fernando Tue, 15 Feb 2011 14:24:28 +0000 http://probablyirrelevant.org/?p=91#comment-13322 Miles, how is that different from handling click spam in general? Miles, how is that different from handling click spam in general?

]]> Comment on Explicit negative feedback comes to the web…somewhat by miles http://probablyirrelevant.org/2011/02/explicit-negative-feedback-comes-to-the-web-somewhat/comment-page-1/#comment-13320 miles Tue, 15 Feb 2011 14:04:04 +0000 http://probablyirrelevant.org/?p=91#comment-13320 When I read about this, the first question that came to mind here is, how can we marshal negative feedback without risking gaming of the system? i.e. Most black-hat approaches aim to raise a site in organic search. Here we invite the opposite problem. If I'm a company with a direct competitor in my search space, a naive negative fb mechanism would allow me to 'bomb' that competitor's results. So clearly, any production-level use of these data will require a resourceful approach... a new sub-problem in the adversarial IR domain? When I read about this, the first question that came to mind here is, how can we marshal negative feedback without risking gaming of the system? i.e. Most black-hat approaches aim to raise a site in organic search. Here we invite the opposite problem. If I’m a company with a direct competitor in my search space, a naive negative fb mechanism would allow me to ‘bomb’ that competitor’s results.

So clearly, any production-level use of these data will require a resourceful approach… a new sub-problem in the adversarial IR domain?

]]> Comment on Explicit negative feedback comes to the web…somewhat by Jon http://probablyirrelevant.org/2011/02/explicit-negative-feedback-comes-to-the-web-somewhat/comment-page-1/#comment-13311 Jon Tue, 15 Feb 2011 01:47:51 +0000 http://probablyirrelevant.org/?p=91#comment-13311 This is certainly a feature I appreciate, and I'd be interested to see whether or not it plays into a larger plan of modeling true negative feedback. I'd imagine for now they're just casting the net to see what kind of data they pull in. They do seem to be paying attention to Blekko's approach - this is the only Blekko feature I found personally compelling. This is certainly a feature I appreciate, and I’d be interested to see whether or not it plays into a larger plan of modeling true negative feedback. I’d imagine for now they’re just casting the net to see what kind of data they pull in.

They do seem to be paying attention to Blekko’s approach – this is the only Blekko feature I found personally compelling.

]]> Comment on Finding relevance judgements in the wild by Webhosting http://probablyirrelevant.org/2009/04/finding-relevance-judgements-in-the-wild/comment-page-1/#comment-10704 Webhosting Tue, 05 Oct 2010 11:19:27 +0000 http://probablyirrelevant.org/?p=61#comment-10704 I am surprised that out of a large sample of 17k you only selected 48 threads that were relevant to your study. Each test could of course produce different results next time with different criteria. So further experiments may give different results. I am surprised that out of a large sample of 17k you only selected 48 threads that were relevant to your study. Each test could of course produce different results next time with different criteria. So further experiments may give different results.

]]> Comment on Query logs and information retrieval research by Lee S Jensen http://probablyirrelevant.org/2010/06/query-logs-and-information-retrieval-research/comment-page-1/#comment-9424 Lee S Jensen Tue, 22 Jun 2010 14:21:52 +0000 http://probablyirrelevant.org/?p=76#comment-9424 AOL got in big trouble when it released its search log. This was because it violated privacy issues. However, with a vertical search system the logs don't generally have that concern. Being able to track the search path of person when they are only looking at genealogy (www.ancestry.com) or cameras (www.dpreview.com) doesn't really tell you anything valuable about a particular person. Focusing on getting search logs from these types of companies should be our focus. AOL got in big trouble when it released its search log. This was because it violated privacy issues. However, with a vertical search system the logs don’t generally have that concern. Being able to track the search path of person when they are only looking at genealogy (www.ancestry.com) or cameras (www.dpreview.com) doesn’t really tell you anything valuable about a particular person. Focusing on getting search logs from these types of companies should be our focus.

]]> Comment on Query logs and information retrieval research by neelannair http://probablyirrelevant.org/2010/06/query-logs-and-information-retrieval-research/comment-page-1/#comment-9217 neelannair Wed, 09 Jun 2010 14:57:38 +0000 http://probablyirrelevant.org/?p=76#comment-9217 Valid points. But I guess query logs are a huge investment for most search engines. It forms part of their competitive advantage over other search engines. I'd like to propose a time-out clause for search engine query logs. They could release query logs more than 'n' years old for a license and fee. So Universities can give open access to researchers to work with them. As far as the Lemur Project was concerned, I came to know about it in the news that it had been discontinued. There is such a thing as publicity, I guess. Wonder if they tied up with existing search engines or something to promote the cause. Valid points.

But I guess query logs are a huge investment for most search engines. It forms part of their competitive advantage over other search engines.

I’d like to propose a time-out clause for search engine query logs. They could release query logs more than ‘n’ years old for a license and fee. So Universities can give open access to researchers to work with them.

As far as the Lemur Project was concerned, I came to know about it in the news that it had been discontinued. There is such a thing as publicity, I guess. Wonder if they tied up with existing search engines or something to promote the cause.

]]>