As an exercise, I scanned the list of titles from SIGIR 2001 and created a poll to see which papers readers would nominate for an informal SIGIR 2011 Test of Time award.  The poll can be found here.  Three votes per person.  Poll closes on July 24, 2011.

Update: Looks like I did not read the fine print of the polling site close enough and the polls will close on July 24, 2011 or when 100 votes have been received, whichever comes first.  Currently at 63 votes total.  I did specify “informal”, didn’t I?

Update: The poll closed over the weekend and the top three papers captured 50% of the votes,

1. “Relevance based language models”, Victor Lavrenko, W. Bruce Croft (24.39%; citations: 252/ACM,618/G)
2. “A statistical learning learning model of text classification for support vector machines”, Thorsten Joachims (15.85%; citations: 60/ACM,215/G)
3. “A study of smoothing methods for language models applied to Ad Hoc information retrieval”, Chengxiang Zhai, John Lafferty (13.41%; citations: 281/ACM,709/G)

The best paper at SIGIR 2001 was “Temporal summaries of new topics”, James Allan, Rahul Gupta, Vikas Khandelwal (1.22%; citations: 43/ACM,132/G). I cannot find an easy to get the most cited paper at the conference.

The goals of the CIFellows Project are to retain new Ph.D. scholars in research and teaching during challenging economic times, while also supporting intellectual renewal and diversity in the computing fields at U.S. organizations. A total of 107 Ph.D.s have been supported through the program since 2009 (see the box at right for more details).

These CIFellows have received outstanding research and teaching enrichment experiences, and several have landed permanent positions (including tenure-track faculty appointments) in academia and industry as a result of their experiences.

CRA/CCC will make awards for the 2011-12 academic year. The exact number of awards is contingent upon the quality of applications received as well as the outcome of a proposal for funding that we have submitted.

Fellowships support “positions at universities, industrial research laboratories, and other organizations that are pursuing innovation in computing and its positive impact on society.”

The deadline is May 31, 2011.

If you didn’t see the Jeopardy! match, check out the practice match, interview with Professor Nyberg, Hideki and Nico, and background on IBM’s site.

Probably Irrelevant: What role does IR play in a QA system?

Nico Schlaefer: Watson and other state-of-the-art QA systems find answers in unstructured text, which is indexed and searched with IR systems. From an architectural point of view, QA applications are often build on top of an IR system – they take a natural language question and transform it into a query that can be handled by the retrieval engine, submit the query and get the search results, and then further process these results by extracting answers and scoring them. So IR plays a key role in question answering, and the performance of a QA system highly depends on the quality of the search results.

PI: What are good characteristics of an IR system for QA?

NS: In QA, it is quite common to generate relatively complicated queries that include term weights and proximity operators. Some systems also pre-annotate their sources with syntactic or semantic information and formulate constrained queries that leverage these annotations. In addition, QA systems often do not retrieve whole documents but shorter passages comprising just a few sentences. To be suitable for QA, an IR system should provide a rich query language, support annotations on the source documents, and allow QA systems to retrieve search results of different granularities.

PI: Where does IR fail with respect to QA?

NS: IR often fails if there is little relevant information for a given question in the sources. The question and relevant documents may use different terminology, which makes it hard for the IR system to retrieve useful text. Query expansion or pseudo-relevance feedback can help to some extent, but often these techniques do not consistently improve performance, and some QA systems only use them as a fallback solution if an initial search does not return anything useful. Obviously, these methods are not going to help if the answer to a question is not in the sources. We developed a different approach – statistical source expansion – which overcomes some of these issues by augmenting existing sources with more relevant information and by increasing semantic redundancy.

PI: What kinds of resources are expanded?

NS: We focused on sources we found most useful for answering Jeopardy! questions and also question from TREC QA evaluations. These include encyclopedias (such as Wikipedia) and dictionaries (such as Wiktionary). More recently, we also experimented with the ClueWeb09 corpus, a large web crawl created at CMU which comprises about 12 TB of English web pages.

PI: What techniques and/or tools are you using to identify topics to expand?

NS: This depends on the sources. For example, when expanding an encyclopedia or a dictionary, we consider each document as a candidate topic. We can then sort the topics by some measure of popularity and focus on expanding the most popular ones. This approach is based on the assumption that Jeopardy! questions (and also questions in most other QA tasks, such as TREC) tend to ask about popular topics, so we get the largest performance gain out of expanding those topics. When expanding other sources that are not organized by topics, such as web crawls or newswire corpora, more sophisticated topic detection techniques become necessary. For example, the most popular topics can be identified using named entity recognizers, statistical methods or dictionaries of known topics.

PI: How do you estimate relevance to those topics?

NS: We use a statistical model that combines a variety of features to estimate the topicality and textual quality of text passages. For example, one of the topicality features is a likelihood ratio estimated with language models. A topic model is trained using the seed document we’re expanding or related web pages retrieved for that seed, and a background model is trained on a large collection of text. The ratio of the likelihoods of a text passage under the topic model and the background model is a good indicator of topicality. Textual quality can, for example, be estimated using dictionaries of known words and n-grams. We also look at simple surface features of text passages, such as the length of a passage and its offset in the source document.

PI: Could you give an example of a question that is helped by source expansion?

NS: Here is a question for which source expansion helped:

What is the name of the rare neurological disease with symptoms such as: involuntary movements (tics), swearing, and incoherent vocalizations (grunts, shouts, etc.)?

This is a question from the TREC 8 evaluation [pdf], but if written as a statement (”This rare neurological disease has symptoms such as …”) I think it could also pass as a Jeopardy! question. The answer is “Tourette syndrome”.

We first tried to answer this question using Wikipedia as a source, and there is indeed an article about “Tourette syndrome” in our copy of Wikipedia, but unfortunately it doesn’t mention most of the keywords in the question and Watson wasn’t able to get the answer. We then expanded Wikipedia, and “Tourette syndrome” was one of the topics that was automatically selected. The expanded article contains the following text passages which, by the way, all come from different websites:

• Rare neurological disease that causes repetitive motor and vocal tics
• The first symptoms usually are involuntary movements (tics) of the face, arms, limbs or trunk.
• Tourette’s syndrome (TS) is a neurological disorder characterized by repetitive, stereotyped, involuntary movements and vocalizations called tics.
• The person afflicted may also swear or shout strange words, grunt, bark or make other loud sounds.

These passages jointly almost perfectly cover the question keywords. I think the only content word that is not in there is “incoherent”. This made it very easy for Watson to find the answer.

• $5,000 unrestricted research seed funding which can be used for conference fees and travel, lab materials, professional society membership dues, etc.
• Exclusive access to select Yahoo! datasets
• The unique opportunity to collaborate with our industry-leading scientists
• An invitation to this summer’s exclusive Key Scientific Challenges Graduate Student Summit where you’ll join the top minds in academia and industry to present your work, discuss research trends and jointly develop revolutionary approaches to fundamental problems

Deadline is March 11, 2011.

G is upfront about collecting the data,

If installed, the extension also sends blocked site information to Google, and we will study the resulting feedback and explore using it as a potential ranking signal for our search results.

Hopefully this means users are starting to get a better idea of how data flows and is exploited by modern information providers.

Does anyone know of publications using this proprietary data? I mean the whole thing, not just samples we all have access to in our local networks.

A few more questions for those with the data:

• how much does the social network data help in recognizer performance?  other profile data?
• if you suppress all image data for an individual, can you still recognize them with non-random accuracy?
• can you infer any of the structured content in a profile from image data?
• have any companies or government organizations asked you for access to this data?

No need to share the data, just the results.

UPDATE: Facebook (now?) has the following Privacy setting,

Suggest photos of me to friends

When photos look like me, suggest my name

“…this study estimates that TREC’s existence was responsible for approximately one-third of an improvement of more than 200% in web search products that was observed between 1999 and 2009.”

More here.

• A comparison of general vs personalized affective models for the prediction of topical relevance, I. Arapakis, K. Athanasakos, J. Jose
• Assessing the Scenic Route: Measuring the Value of Search Trails in Web Logs, R. White, J. Huang
• Caching Search Engine Results over Incremental Indices, F. Junqueira, R. Blanco, E. Bortnikov, R. Lempel, L. Telloli, H. Zaragoza
• Comparing the Sensitivity of Information Retrieval Metrics, F. Radlinski, N. Craswell
• Extending Average Precision to Graded Relevance Judgments, S. Robertson, E. Kanoulas, E. Yilmaz
• Information Based Model for ad hoc information retrieval, S. Clinchant, E. Gaussier
• Multi-style language model for web scale information retrieval, K. Wang, J. Gao, X. Li
• Properties of Optimally Weighted Data Fusion in CBMIR, P. Wilkins, A. Smeaton

The goal of this project is to create a database of web search activity that will be provided to the information retrieval research community to use on current and future information retrieval research projects.

To accomplish this, the Lemur Project developed a toolbar to be voluntarily installed by users. After a year of data collection, the project has been aborted,

Given that we have gathered the equivalent of less than 6 seconds of Google traffic (assuming 500 million queries per day) in one year, we have decided to terminate the project.

This is pretty depressing news. Admittedly, part of this depression originates from my guilt over not having contributed to the project myself. However, a more substantial part stems from the potential this data set had to be groundbreaking, perhaps similar to the release of the first Tipster collections. Although this was way before my time, I imagine the sudden release of a large, public corpus resulted in a tremendous amount of activity and excitement.

Information retrieval research has had large collections of documents for a few decades now. We evaluate on a few hundred queries and publish results. With some exceptions, the majority of interest in the field has focused on scaling up corpora. As a result, we have rich set of tools to analyze and retrieve documents from large corpora.

There are two things missing from this model: a rich stream of queries coming into the system and a rich stream of interactions between users and documents. Our friends in the CHI and information science communities have been doing a great job with understanding the important factors involved in user behavior on laboratory scale. However, I’m going to draw an analogy here between small scale user studies for IR and document-level NLP analysis for IR that may raise a few eyebrows. I believe that many IR researchers would argue that, given the choice between a corpus-driven approaches and NLP approaches to IR, they would opt for more data. This is despite the rich analysis NLP can provide. Similarly, I believe that the fine-grained analysis provided by laboratory studies may be less important than very large scale analysis of user behavior. Of course, both the results about NLP for IR and the claim about laboratory experiments are based on relatively limited experiments (e.g. small sets of queries). We should, as a community, continue research in all of these directions.

Having said this, let’s consider some motivations for web query logs and IR research,

Claim 1. Web query logs will help with the contribution to web search research.

There is no doubt that query logs are important for any search engine, web or otherwise. However, query logs are only one of the many sources of interaction data available in production. There are many, many other signals which can be effectively exploited for query understanding and document ranking. In my opinion, outside of starting its own web search engine, academia will always be scurrying to catchup to industry’s data sources.

I convinced myself a few years ago that the resources required to build and maintain a web search engine may never exist in academia. This is not to say that academic IR researchers should give up on having impact on web search engines. IR research several decades old continues to impact modern search engine design. What needs to be determined is how the current academic IR researchers can more directly address the problems confronted by web search companies. I personally believe that a tight coupling between academic and industrial research labs needs to exist. This could be accomplished in a number of ways.

1. add value to an existing search engine’s interface. If search engines provide ranker APIs, academics can develop new interfaces which may attract users and, as a result, interaction data.
2. teach the IR fundamentals during the academic year/perform intense interaction during the summer during internships or other collaborations. I am most familiar with Yahoo’s Key Scientific Challenges Fellowships and Faculty Engagement Grants. Similar programs exist at other web search engines.
3. develop high-quality, public web search engine simulators which provide students/researchers with the ability to test algorithms in silico. Our SIGIR 2009 paper made extensive use of simulation whose parameters were grounded in real world data. Systems research in computer architecture or computer networking have adopted this approach for a while. SIGIR 2010 will be hosting a workshop on simulated interaction.

No doubt there are many, many other alternatives.

Claim 2. Web query logs will help with the contribution to production search research.

As stated earlier, IR research has looked at the document side for many, many problems. This research has benefited web search as well as search in other domains such as legal, news, and enterprise search.

User behavior data improved production web search engines; user behavior data will no doubt improve production non-web search engines. Just as with web search though, this data does not exist in academia.

I believe, though, that the barrier to entry for non-web/vertical search engines is somewhat lower. The collections are smaller and manageable. At the same time, document representations can be richer for verticals, interaction is less constrained, and, as a result, the potential for attracting users may be higher than with portal web search engines.

If an academic institution maintained a domain-specific production search engine, academic research could become more relevant to industrial search engines. For example, academic institutions would easily be able to publish about query logs, interaction, large scale adaptation, and online learning with large scale real world data.  One important, unresolved question is how to come to terms with experimental reproducibility and production data which is often closed due to privacy reasons.

Academic IR research will continue to contribute to general IR research. Students trained in IR fundamentals will continue to be strong candidates for research and development in production search companies. I believe that there is room for greater impact. How that happens remains to be seen.