“…this study estimates that TREC’s existence was responsible for approximately one-third of an improvement of more than 200% in web search products that was observed between 1999 and 2009.”

More here.

• A comparison of general vs personalized affective models for the prediction of topical relevance, I. Arapakis, K. Athanasakos, J. Jose
• Assessing the Scenic Route: Measuring the Value of Search Trails in Web Logs, R. White, J. Huang
• Caching Search Engine Results over Incremental Indices, F. Junqueira, R. Blanco, E. Bortnikov, R. Lempel, L. Telloli, H. Zaragoza
• Comparing the Sensitivity of Information Retrieval Metrics, F. Radlinski, N. Craswell
• Extending Average Precision to Graded Relevance Judgments, S. Robertson, E. Kanoulas, E. Yilmaz
• Information Based Model for ad hoc information retrieval, S. Clinchant, E. Gaussier
• Multi-style language model for web scale information retrieval, K. Wang, J. Gao, X. Li
• Properties of Optimally Weighted Data Fusion in CBMIR, P. Wilkins, A. Smeaton

The goal of this project is to create a database of web search activity that will be provided to the information retrieval research community to use on current and future information retrieval research projects.

To accomplish this, the Lemur Project developed a toolbar to be voluntarily installed by users. After a year of data collection, the project has been aborted,

Given that we have gathered the equivalent of less than 6 seconds of Google traffic (assuming 500 million queries per day) in one year, we have decided to terminate the project.

This is pretty depressing news. Admittedly, part of this depression originates from my guilt over not having contributed to the project myself. However, a more substantial part stems from the potential this data set had to be groundbreaking, perhaps similar to the release of the first Tipster collections. Although this was way before my time, I imagine the sudden release of a large, public corpus resulted in a tremendous amount of activity and excitement.

Information retrieval research has had large collections of documents for a few decades now. We evaluate on a few hundred queries and publish results. With some exceptions, the majority of interest in the field has focused on scaling up corpora. As a result, we have rich set of tools to analyze and retrieve documents from large corpora.

There are two things missing from this model: a rich stream of queries coming into the system and a rich stream of interactions between users and documents. Our friends in the CHI and information science communities have been doing a great job with understanding the important factors involved in user behavior on laboratory scale. However, I’m going to draw an analogy here between small scale user studies for IR and document-level NLP analysis for IR that may raise a few eyebrows. I believe that many IR researchers would argue that, given the choice between a corpus-driven approaches and NLP approaches to IR, they would opt for more data. This is despite the rich analysis NLP can provide. Similarly, I believe that the fine-grained analysis provided by laboratory studies may be less important than very large scale analysis of user behavior. Of course, both the results about NLP for IR and the claim about laboratory experiments are based on relatively limited experiments (e.g. small sets of queries). We should, as a community, continue research in all of these directions.

Having said this, let’s consider some motivations for web query logs and IR research,

Claim 1. Web query logs will help with the contribution to web search research.

There is no doubt that query logs are important for any search engine, web or otherwise. However, query logs are only one of the many sources of interaction data available in production. There are many, many other signals which can be effectively exploited for query understanding and document ranking. In my opinion, outside of starting its own web search engine, academia will always be scurrying to catchup to industry’s data sources.

I convinced myself a few years ago that the resources required to build and maintain a web search engine may never exist in academia. This is not to say that academic IR researchers should give up on having impact on web search engines. IR research several decades old continues to impact modern search engine design. What needs to be determined is how the current academic IR researchers can more directly address the problems confronted by web search companies. I personally believe that a tight coupling between academic and industrial research labs needs to exist. This could be accomplished in a number of ways.

1. add value to an existing search engine’s interface. If search engines provide ranker APIs, academics can develop new interfaces which may attract users and, as a result, interaction data.
2. teach the IR fundamentals during the academic year/perform intense interaction during the summer during internships or other collaborations. I am most familiar with Yahoo’s Key Scientific Challenges Fellowships and Faculty Engagement Grants. Similar programs exist at other web search engines.
3. develop high-quality, public web search engine simulators which provide students/researchers with the ability to test algorithms in silico. Our SIGIR 2009 paper made extensive use of simulation whose parameters were grounded in real world data. Systems research in computer architecture or computer networking have adopted this approach for a while. SIGIR 2010 will be hosting a workshop on simulated interaction.

No doubt there are many, many other alternatives.

Claim 2. Web query logs will help with the contribution to production search research.

As stated earlier, IR research has looked at the document side for many, many problems. This research has benefited web search as well as search in other domains such as legal, news, and enterprise search.

User behavior data improved production web search engines; user behavior data will no doubt improve production non-web search engines. Just as with web search though, this data does not exist in academia.

I believe, though, that the barrier to entry for non-web/vertical search engines is somewhat lower. The collections are smaller and manageable. At the same time, document representations can be richer for verticals, interaction is less constrained, and, as a result, the potential for attracting users may be higher than with portal web search engines.

If an academic institution maintained a domain-specific production search engine, academic research could become more relevant to industrial search engines. For example, academic institutions would easily be able to publish about query logs, interaction, large scale adaptation, and online learning with large scale real world data.  One important, unresolved question is how to come to terms with experimental reproducibility and production data which is often closed due to privacy reasons.

Academic IR research will continue to contribute to general IR research. Students trained in IR fundamentals will continue to be strong candidates for research and development in production search companies. I believe that there is room for greater impact. How that happens remains to be seen.

Update: Here is the SIGIR Digital Museum of Information Retrieval Research, including those SMART reports.

For lack of a better term, I think of this phenomenon as “micro information retrieval” (micro-IR).

By micro-IR I mean the practice of farming information needs out across multiple applications.  Each of these micro-IR applications is built around a tightly constrained problem space, and I think it’s this constraint that makes micro-IR interesting.

A couple of examples (apologies for any appearance of commercial endorsement; none intended):

• the yelp app: find, say, restaurants near me
• loopt: find friends near me
• Barnes and Noble app: find info on the book in this photo I took
• shazam: find the song that is playing into the iphone mic.

These examples swing close to simple database lookups.  But if we take a longer view, a more interesting dynamic comes up.  The apps are simple because each one solves a problem that is tightly constrained, answering a question that would involve complicated interaction in its absence.

By way of a few more examples, I am currently developing an app that answers the question: how many gallons of oil would it take to prepare a given recipe?  The app then ranks candidate recipes in increasing order of petroleum consumption.

And it’s not the case that these sorts of interactions are limited to mobile devices.  Thanks to Gene Golovchinsky for pointing me towards Blueprint an Eclipse plugin that allows users to search for code snippets from within their IDE, leveraging Flex syntax to finesse the search.

Trying to lasso these examples together in efforts to triangulate on what micro-IR actually is, I’ll note a few overarching commonalities that I see here:

1. In ad hoc (text) IR a principal intellectual challenge lies in modeling ‘aboutness.’  In micro-IR settings, the creativity comes into play in posing a useful (and tractable) question to answer.  The engineering comes easily after that.
2. The constrained nature of micro-IR applications leads to a lightweight articulation of information need.  There is a tight coupling here between task, query, and the unit of retrieval, a dynamic that I think is compelling.  Pushing this a bit farther, we might consider the simple act of choosing to use a particular application from those apps on a user’s palette as part of the information need expression.
3. The tight coupling of task to data to ‘query’ enables a strong contextual element to inform the interaction.  Context constitutes the foreground of the micro-IR interaction.

I don’t want to overstate the distinction between micro- and macro-IR.  Of course applications fall along a spectrum of their similarity to the modalities I’ve laid out here.  But I do think that being aware of micro-IR system characteristics is worthwhile.  Aside from an inherent innovation to how people interact with information, micro-IR opens the door to small-scale developers gaining a wide audience (i.e. the barrier to entry is low).  And concomitant with this is the new monetization model at work in the app store.

I hope readers will comment on this: is micro-IR something at all?  Is it actually related to IR?  How might we turn our eye to micro-IR with respect to generating bona fide research?  Surely there are better example systems than those I’ve listed…

There’s no existing IR test collection for such a task, although some similar datasets do exist. For various reasons we weren’t able to create a traditional test collection, with user-issued queries and deep pools of relevance judgements. But, this particular dataset and possibly other online dialog archives can be mined to produce a ready-made IR test collection.

The users of the online forum we’ve been looking at frequently include links in their forum posts — often to previous messages and threads in the same forum. These links are sometimes in response to a new user’s question, and refer the user to a previous instance of the same (or similar) question and an answer contributed by another user. Here’s a few examples to illustrate my point. This interaction among forum users can be used as a form of query/relevance judgement pair. See the paper for a few more details on how we characterize the presence of a question-post/answer-link pair.

This type of test collection creation does have some distinct advantages over the typical retrieval test collections used at TREC. First, the queries represent real information needs of real users of the online forum. Many TREC queries are pulled from search engine logs, but frequently (as in the Blog Track’s Feed Distillation task) the queries are invented by participants or assessors. The information needs present in the online forum posts are much more verbose than typical keyword queries on a web search engine, providing a retrieval system more evidence with which to use in relevance scoring. The “relevance judgement”, provided by another forum user linking to a previous thread, also presents in-situ relevance information — sensitive not only to the original question, but also to the overall nature of the forum and the time when the question was asked.

There are several drawbacks inherent in this type of corpus creation, most importantly with regard to the exhaustiveness of the relevance assessment. Typically in TREC-style collection development, ranked results from several retrieval systems are pooled and those pooled documents are assessed for relevance. When the systems’ output is sufficiently diverse and relevance assessment is sufficiently deep, this produces a reasonably complete relevance assessment for each query — if a relevant document is in the collection, it would most likely be retrieved by one of the systems and be judged by being admitted into the pool. The method of collecting relevance judgements we use in our SIGIR poster, on the other hand, will not produce anything close to an exhaustive set of relevant threads. In the great majority of cases, only a single thread is linked to in a subsequent reply message. There is no guarantee that this thread is the best or only relevant thread in the collection. For this reason, we must take care not to assume non-judged threads are necessarily irrelevant.

There are plenty of datasets that seem to be ready-made for classification or regression tasks, without any need for annotation — for example the classic 20 newsgroups for text classification and Yahoo! Answers for a number of prediction tasks. For relevance ranking, however, I haven’t seen any ready-made datasets with real relevance judgements, as opposed to noisy interaction indicators such as click-through statistics. Conversation archives like the one we use offer one way to mine behavioral data for relevance judgements, offering ground-truth preferable in many ways to post-hoc relevance assessment.

PAPERS

The program committee reviewed 494 full paper submissions and accepted 78, about a 16% acceptance rate.

In 2008, 497 submitted, 85 accepted, about a 17% acceptance rate.

POSTERS

The program committee reviewed 256 poster submissions and accepted 86, about a 34% acceptance rate.

In 2008, 173 submitted, 91 accepted, about a 53% acceptance rate.

UPDATE

Accepted papers published.

• SIGIR 2009 Homepage
• SIGIR 2009 Facebook Event

Important Dates:

Community generated content has been all the rage for a few years: blogs, Wikipedia, online forums, twitter, Yahoo! Answers, and the list goes on. Many of these generate a large volume of archived data — some in the form of more or less polished documents, like a blog post or Wikipedia article; others, like twitter, are snippets of an often one-sided conversation and broadcast messages.

From the IR researcher’s perspective, is it worth studying these artifacts of “social media”? Is there something that distinguishes these from other document collections? If so, how can we leverage that distinction in our retrieval models? This post aims to answer a couple of these questions and hopefully bring up a few more.

First and foremost, we need to identify whether there is value in providing access to artifacts of social media.  Some, like twitter, seem to be mostly ephemeral, only (generally) interesting in the moment and quickly fading from view.  Even the twitter search engine advertises: “See what’s happening — right now” and the results (as far as I can tell) are only ranked chronologically.  

Many other types of social media — some existing long before Web 2.0 was born — can be real treasure-troves of information. There exists an online forum, public mailing lists, newsgroup or message board for virtually every special interest group under the sun — from gardening, to home-brewing, to apple computers. These are often heavily trafficked, populated with real subject matter experts, and host a rich information exchange. I would argue that the content created through these social media outlets present an enormous value to searchers, and information retrieval research has a lot to contribute in this corner of social media.

What makes these document collections different than what has been previously studied? Can we just treat them the same as web pages? Or do they need special consideration?

In many of these collections, the unit of retrieval — what we consider a document — is not fixed, but rather dependent on the task. Consider online forums, often organized into topical sub-forums, which in turn are organized into conversation threads of individual posts. Some information needs many only require a single post as a result, some require the context of the full conversation thread, and others may need to retrieve a pertinent sub-forum.

These collections often offer another orthogonal axis of retrieval — the author. In highly trafficked message boards and mailing lists, tens or hundreds of thousands of users with varying levels of expertise contribute to the conversation. One may wish to find subject matter experts to address a question to, or favor message threads with contributions from those more likely to know the answer.

These factors, of course, are not entirely unique to social media search, and have to some degree been addressed in previous research. This question of identifying the granularity of the unit of retrieval has been addressed at the document level (for example in XML element retrieval at INEX), but not so much at the collection level. Resource ranking in federated search and cluster-based retrieval bear some resemblance to the selection of a topical sub-collection, such as a sub-forum ranking. Author-ranking has also been studied at TREC in the Blog and Enterprise Tracks. But, each of these have been studied in isolation, without much regard to the interaction between the different aspects of the collection. To my knowledge, no IR testbeds exist that contain the rich collection structure offered in these types of social media.

This, in my mind, is the real promise of research in search over social media. These collections provide multiple levels of organizational granularity, different axes of organization, multiple types of searchable objects, and relations among those objects.  I predict that this will be an interesting and fertile direction of information retrieval research — pushing the systems to support more sophisticated multi-dimensional indexing and extending existing retrieval models to handle rich relationships between documents.