“…this study estimates that TREC’s existence was responsible for approximately one-third of an improvement of more than 200% in web search products that was observed between 1999 and 2009.”

More here.

• A comparison of general vs personalized affective models for the prediction of topical relevance, I. Arapakis, K. Athanasakos, J. Jose
• Assessing the Scenic Route: Measuring the Value of Search Trails in Web Logs, R. White, J. Huang
• Caching Search Engine Results over Incremental Indices, F. Junqueira, R. Blanco, E. Bortnikov, R. Lempel, L. Telloli, H. Zaragoza
• Comparing the Sensitivity of Information Retrieval Metrics, F. Radlinski, N. Craswell
• Extending Average Precision to Graded Relevance Judgments, S. Robertson, E. Kanoulas, E. Yilmaz
• Information Based Model for ad hoc information retrieval, S. Clinchant, E. Gaussier
• Multi-style language model for web scale information retrieval, K. Wang, J. Gao, X. Li
• Properties of Optimally Weighted Data Fusion in CBMIR, P. Wilkins, A. Smeaton

The goal of this project is to create a database of web search activity that will be provided to the information retrieval research community to use on current and future information retrieval research projects.

To accomplish this, the Lemur Project developed a toolbar to be voluntarily installed by users. After a year of data collection, the project has been aborted,

Given that we have gathered the equivalent of less than 6 seconds of Google traffic (assuming 500 million queries per day) in one year, we have decided to terminate the project.

This is pretty depressing news. Admittedly, part of this depression originates from my guilt over not having contributed to the project myself. However, a more substantial part stems from the potential this data set had to be groundbreaking, perhaps similar to the release of the first Tipster collections. Although this was way before my time, I imagine the sudden release of a large, public corpus resulted in a tremendous amount of activity and excitement.

Information retrieval research has had large collections of documents for a few decades now. We evaluate on a few hundred queries and publish results. With some exceptions, the majority of interest in the field has focused on scaling up corpora. As a result, we have rich set of tools to analyze and retrieve documents from large corpora.

There are two things missing from this model: a rich stream of queries coming into the system and a rich stream of interactions between users and documents. Our friends in the CHI and information science communities have been doing a great job with understanding the important factors involved in user behavior on laboratory scale. However, I’m going to draw an analogy here between small scale user studies for IR and document-level NLP analysis for IR that may raise a few eyebrows. I believe that many IR researchers would argue that, given the choice between a corpus-driven approaches and NLP approaches to IR, they would opt for more data. This is despite the rich analysis NLP can provide. Similarly, I believe that the fine-grained analysis provided by laboratory studies may be less important than very large scale analysis of user behavior. Of course, both the results about NLP for IR and the claim about laboratory experiments are based on relatively limited experiments (e.g. small sets of queries). We should, as a community, continue research in all of these directions.

Having said this, let’s consider some motivations for web query logs and IR research,

Claim 1. Web query logs will help with the contribution to web search research.

There is no doubt that query logs are important for any search engine, web or otherwise. However, query logs are only one of the many sources of interaction data available in production. There are many, many other signals which can be effectively exploited for query understanding and document ranking. In my opinion, outside of starting its own web search engine, academia will always be scurrying to catchup to industry’s data sources.

I convinced myself a few years ago that the resources required to build and maintain a web search engine may never exist in academia. This is not to say that academic IR researchers should give up on having impact on web search engines. IR research several decades old continues to impact modern search engine design. What needs to be determined is how the current academic IR researchers can more directly address the problems confronted by web search companies. I personally believe that a tight coupling between academic and industrial research labs needs to exist. This could be accomplished in a number of ways.

1. add value to an existing search engine’s interface. If search engines provide ranker APIs, academics can develop new interfaces which may attract users and, as a result, interaction data.
2. teach the IR fundamentals during the academic year/perform intense interaction during the summer during internships or other collaborations. I am most familiar with Yahoo’s Key Scientific Challenges Fellowships and Faculty Engagement Grants. Similar programs exist at other web search engines.
3. develop high-quality, public web search engine simulators which provide students/researchers with the ability to test algorithms in silico. Our SIGIR 2009 paper made extensive use of simulation whose parameters were grounded in real world data. Systems research in computer architecture or computer networking have adopted this approach for a while. SIGIR 2010 will be hosting a workshop on simulated interaction.

No doubt there are many, many other alternatives.

Claim 2. Web query logs will help with the contribution to production search research.

As stated earlier, IR research has looked at the document side for many, many problems. This research has benefited web search as well as search in other domains such as legal, news, and enterprise search.

User behavior data improved production web search engines; user behavior data will no doubt improve production non-web search engines. Just as with web search though, this data does not exist in academia.

I believe, though, that the barrier to entry for non-web/vertical search engines is somewhat lower. The collections are smaller and manageable. At the same time, document representations can be richer for verticals, interaction is less constrained, and, as a result, the potential for attracting users may be higher than with portal web search engines.

If an academic institution maintained a domain-specific production search engine, academic research could become more relevant to industrial search engines. For example, academic institutions would easily be able to publish about query logs, interaction, large scale adaptation, and online learning with large scale real world data.  One important, unresolved question is how to come to terms with experimental reproducibility and production data which is often closed due to privacy reasons.

Academic IR research will continue to contribute to general IR research. Students trained in IR fundamentals will continue to be strong candidates for research and development in production search companies. I believe that there is room for greater impact. How that happens remains to be seen.

Update: Here is the SIGIR Digital Museum of Information Retrieval Research, including those SMART reports.

PAPERS

The program committee reviewed 494 full paper submissions and accepted 78, about a 16% acceptance rate.

In 2008, 497 submitted, 85 accepted, about a 17% acceptance rate.

POSTERS

The program committee reviewed 256 poster submissions and accepted 86, about a 34% acceptance rate.

In 2008, 173 submitted, 91 accepted, about a 53% acceptance rate.

UPDATE

Accepted papers published.

• SIGIR 2009 Homepage
• SIGIR 2009 Facebook Event

Important Dates:

Now, it seems appropriate that the inaugural post of an information retrieval blog should address the subject of “blog search”. Unfortunately, I am dreadfully less qualified than my co-authors to discuss the state of the art. So, I apologize in advance for errors, omissions, or general ridiculousness and lay blame on Kevyn and Jonathan.

Now, when I started “Information Retrieval”, one of the first messages I received was from a senior member of the IR community. He wrote,

Maybe you could blog about why anyone is interested in blogs :-)

I replied,

I’ll keep this in mind when you’re chairing a session on blog search at SIGIR 2010.

I will not identify the original commenter but encourage conference attendees to pay attention in Geneva.

Of course, this comment deserves some thought. One of the issues with blog search is the under-defined taxonomy of queries. The TREC Blog Track defines the following tasks

• blog post retrieval (i.e. “Find me posts about X.”)
• opinion retrieval (i.e. “What do people think about X?”)
• polarity (i.e. “Find me positive posts about X.”)
• feed distillation (i.e. “Find me a blog with a principle, recurring interest in X.”)

One question I hope will be resolved in the comments is where these query types came from. Are they derived from actual blog searchers? Or are they merely contrived by the track organizers while trading pints at the Gaithersburg Marriot? These are questions, not criticisms. I think these are fine tasks but we have to be careful to define queries which are representative of those being issued blog search engines or, more generally, fulfill some desire users have. The problem with a new corpus is that how users interact with it is still not completely developed. What users will actually use these systems? Casual blog readers? Marketers? Political scientists? Sociologists?

The majority of time in an “Introduction to Information Retrieval” course is devoted to modeling documents. And, yes, we have sophisticated models of documents. We decompose individual documents using passages, sentences, or other exploitable structure. We also model the corpus as a whole either explicitly (e.g. cluster-based retrieval, latent semantic indexing, regularization) or implicitly (e.g. pseudo-relevance feedback).

For an information retrieval researcher, a corpus without queries is a corpse. Queries make information retrieval different from unsupervised learning. Also, because they are so short, queries make information retrieval different from traditional text classification. While information retrieval research has focused on ranking documents given a query, prior to the late 1990s, there were very few (published) results on modeling queries in aggregate. However, with the advent of web search engines, there has been a growing body of work on such models. These include descriptive studies of web query frequencies and user clicking behavior as well as models for query similarity and clicking behavior. These results have mainly been presented for web users and queries; I would be very interested in seeing whether the results generalize to non-web search scenarios.

To come back to blog search, I believe we need a better understanding of both the corpus and the queries before defining tasks. Blog corpora exist and are actively being studied. I am less certain about blog queries. One approach would be to inspect query logs to blog search engines for different retrieval scenarios and then improve performance for those scenarios. Of course, some of us are engineers who sometimes desire to build a tool because we believe it would be used. However, if there is a mismatch between what we believe will be useful and what users find useful, then we have wasted time.*

I’ve touched on a lot in this first post and hope it serves as a starting point of discussion. So, welcome to “Probably Irrelevant”.

*I just became aware of a paper to be presented at CIKM entitled “What Should Blog Search Look Like?” which I hope will answer some of these questions.

Editor’s Note: Many thanks to Fernando for authoring our first post.  He couldn’t have chosen a more timely topic, the TREC 2008 Blog Track judgements are underway, Iadh Ounis as recently posted a call for suggestions for the 2009 tasks, Jeff Dalton has an insightful response, and Marti Hearst’s paper is now online.