If you didn’t see the Jeopardy! match, check out the practice match, interview with Professor Nyberg, Hideki and Nico, and background on IBM’s site.

Probably Irrelevant: What role does IR play in a QA system?

Nico Schlaefer: Watson and other state-of-the-art QA systems find answers in unstructured text, which is indexed and searched with IR systems. From an architectural point of view, QA applications are often build on top of an IR system – they take a natural language question and transform it into a query that can be handled by the retrieval engine, submit the query and get the search results, and then further process these results by extracting answers and scoring them. So IR plays a key role in question answering, and the performance of a QA system highly depends on the quality of the search results.

PI: What are good characteristics of an IR system for QA?

NS: In QA, it is quite common to generate relatively complicated queries that include term weights and proximity operators. Some systems also pre-annotate their sources with syntactic or semantic information and formulate constrained queries that leverage these annotations. In addition, QA systems often do not retrieve whole documents but shorter passages comprising just a few sentences. To be suitable for QA, an IR system should provide a rich query language, support annotations on the source documents, and allow QA systems to retrieve search results of different granularities.

PI: Where does IR fail with respect to QA?

NS: IR often fails if there is little relevant information for a given question in the sources. The question and relevant documents may use different terminology, which makes it hard for the IR system to retrieve useful text. Query expansion or pseudo-relevance feedback can help to some extent, but often these techniques do not consistently improve performance, and some QA systems only use them as a fallback solution if an initial search does not return anything useful. Obviously, these methods are not going to help if the answer to a question is not in the sources. We developed a different approach – statistical source expansion – which overcomes some of these issues by augmenting existing sources with more relevant information and by increasing semantic redundancy.

PI: What kinds of resources are expanded?

NS: We focused on sources we found most useful for answering Jeopardy! questions and also question from TREC QA evaluations. These include encyclopedias (such as Wikipedia) and dictionaries (such as Wiktionary). More recently, we also experimented with the ClueWeb09 corpus, a large web crawl created at CMU which comprises about 12 TB of English web pages.

PI: What techniques and/or tools are you using to identify topics to expand?

NS: This depends on the sources. For example, when expanding an encyclopedia or a dictionary, we consider each document as a candidate topic. We can then sort the topics by some measure of popularity and focus on expanding the most popular ones. This approach is based on the assumption that Jeopardy! questions (and also questions in most other QA tasks, such as TREC) tend to ask about popular topics, so we get the largest performance gain out of expanding those topics. When expanding other sources that are not organized by topics, such as web crawls or newswire corpora, more sophisticated topic detection techniques become necessary. For example, the most popular topics can be identified using named entity recognizers, statistical methods or dictionaries of known topics.

PI: How do you estimate relevance to those topics?

NS: We use a statistical model that combines a variety of features to estimate the topicality and textual quality of text passages. For example, one of the topicality features is a likelihood ratio estimated with language models. A topic model is trained using the seed document we’re expanding or related web pages retrieved for that seed, and a background model is trained on a large collection of text. The ratio of the likelihoods of a text passage under the topic model and the background model is a good indicator of topicality. Textual quality can, for example, be estimated using dictionaries of known words and n-grams. We also look at simple surface features of text passages, such as the length of a passage and its offset in the source document.

PI: Could you give an example of a question that is helped by source expansion?

NS: Here is a question for which source expansion helped:

What is the name of the rare neurological disease with symptoms such as: involuntary movements (tics), swearing, and incoherent vocalizations (grunts, shouts, etc.)?

This is a question from the TREC 8 evaluation [pdf], but if written as a statement (”This rare neurological disease has symptoms such as …”) I think it could also pass as a Jeopardy! question. The answer is “Tourette syndrome”.

We first tried to answer this question using Wikipedia as a source, and there is indeed an article about “Tourette syndrome” in our copy of Wikipedia, but unfortunately it doesn’t mention most of the keywords in the question and Watson wasn’t able to get the answer. We then expanded Wikipedia, and “Tourette syndrome” was one of the topics that was automatically selected. The expanded article contains the following text passages which, by the way, all come from different websites:

• Rare neurological disease that causes repetitive motor and vocal tics
• The first symptoms usually are involuntary movements (tics) of the face, arms, limbs or trunk.
• Tourette’s syndrome (TS) is a neurological disorder characterized by repetitive, stereotyped, involuntary movements and vocalizations called tics.
• The person afflicted may also swear or shout strange words, grunt, bark or make other loud sounds.

These passages jointly almost perfectly cover the question keywords. I think the only content word that is not in there is “incoherent”. This made it very easy for Watson to find the answer.

There’s no existing IR test collection for such a task, although some similar datasets do exist. For various reasons we weren’t able to create a traditional test collection, with user-issued queries and deep pools of relevance judgements. But, this particular dataset and possibly other online dialog archives can be mined to produce a ready-made IR test collection.

The users of the online forum we’ve been looking at frequently include links in their forum posts — often to previous messages and threads in the same forum. These links are sometimes in response to a new user’s question, and refer the user to a previous instance of the same (or similar) question and an answer contributed by another user. Here’s a few examples to illustrate my point. This interaction among forum users can be used as a form of query/relevance judgement pair. See the paper for a few more details on how we characterize the presence of a question-post/answer-link pair.

This type of test collection creation does have some distinct advantages over the typical retrieval test collections used at TREC. First, the queries represent real information needs of real users of the online forum. Many TREC queries are pulled from search engine logs, but frequently (as in the Blog Track’s Feed Distillation task) the queries are invented by participants or assessors. The information needs present in the online forum posts are much more verbose than typical keyword queries on a web search engine, providing a retrieval system more evidence with which to use in relevance scoring. The “relevance judgement”, provided by another forum user linking to a previous thread, also presents in-situ relevance information — sensitive not only to the original question, but also to the overall nature of the forum and the time when the question was asked.

There are several drawbacks inherent in this type of corpus creation, most importantly with regard to the exhaustiveness of the relevance assessment. Typically in TREC-style collection development, ranked results from several retrieval systems are pooled and those pooled documents are assessed for relevance. When the systems’ output is sufficiently diverse and relevance assessment is sufficiently deep, this produces a reasonably complete relevance assessment for each query — if a relevant document is in the collection, it would most likely be retrieved by one of the systems and be judged by being admitted into the pool. The method of collecting relevance judgements we use in our SIGIR poster, on the other hand, will not produce anything close to an exhaustive set of relevant threads. In the great majority of cases, only a single thread is linked to in a subsequent reply message. There is no guarantee that this thread is the best or only relevant thread in the collection. For this reason, we must take care not to assume non-judged threads are necessarily irrelevant.

There are plenty of datasets that seem to be ready-made for classification or regression tasks, without any need for annotation — for example the classic 20 newsgroups for text classification and Yahoo! Answers for a number of prediction tasks. For relevance ranking, however, I haven’t seen any ready-made datasets with real relevance judgements, as opposed to noisy interaction indicators such as click-through statistics. Conversation archives like the one we use offer one way to mine behavioral data for relevance judgements, offering ground-truth preferable in many ways to post-hoc relevance assessment.

Community generated content has been all the rage for a few years: blogs, Wikipedia, online forums, twitter, Yahoo! Answers, and the list goes on. Many of these generate a large volume of archived data — some in the form of more or less polished documents, like a blog post or Wikipedia article; others, like twitter, are snippets of an often one-sided conversation and broadcast messages.

From the IR researcher’s perspective, is it worth studying these artifacts of “social media”? Is there something that distinguishes these from other document collections? If so, how can we leverage that distinction in our retrieval models? This post aims to answer a couple of these questions and hopefully bring up a few more.

First and foremost, we need to identify whether there is value in providing access to artifacts of social media.  Some, like twitter, seem to be mostly ephemeral, only (generally) interesting in the moment and quickly fading from view.  Even the twitter search engine advertises: “See what’s happening — right now” and the results (as far as I can tell) are only ranked chronologically.  

Many other types of social media — some existing long before Web 2.0 was born — can be real treasure-troves of information. There exists an online forum, public mailing lists, newsgroup or message board for virtually every special interest group under the sun — from gardening, to home-brewing, to apple computers. These are often heavily trafficked, populated with real subject matter experts, and host a rich information exchange. I would argue that the content created through these social media outlets present an enormous value to searchers, and information retrieval research has a lot to contribute in this corner of social media.

What makes these document collections different than what has been previously studied? Can we just treat them the same as web pages? Or do they need special consideration?

In many of these collections, the unit of retrieval — what we consider a document — is not fixed, but rather dependent on the task. Consider online forums, often organized into topical sub-forums, which in turn are organized into conversation threads of individual posts. Some information needs many only require a single post as a result, some require the context of the full conversation thread, and others may need to retrieve a pertinent sub-forum.

These collections often offer another orthogonal axis of retrieval — the author. In highly trafficked message boards and mailing lists, tens or hundreds of thousands of users with varying levels of expertise contribute to the conversation. One may wish to find subject matter experts to address a question to, or favor message threads with contributions from those more likely to know the answer.

These factors, of course, are not entirely unique to social media search, and have to some degree been addressed in previous research. This question of identifying the granularity of the unit of retrieval has been addressed at the document level (for example in XML element retrieval at INEX), but not so much at the collection level. Resource ranking in federated search and cluster-based retrieval bear some resemblance to the selection of a topical sub-collection, such as a sub-forum ranking. Author-ranking has also been studied at TREC in the Blog and Enterprise Tracks. But, each of these have been studied in isolation, without much regard to the interaction between the different aspects of the collection. To my knowledge, no IR testbeds exist that contain the rich collection structure offered in these types of social media.

This, in my mind, is the real promise of research in search over social media. These collections provide multiple levels of organizational granularity, different axes of organization, multiple types of searchable objects, and relations among those objects.  I predict that this will be an interesting and fertile direction of information retrieval research — pushing the systems to support more sophisticated multi-dimensional indexing and extending existing retrieval models to handle rich relationships between documents.

IR is far, far more than a branch of computer science, concerned primarily with issues of algorithms, computers, and computing.

Tefko Saracevic, Acceptance address for the 1997 Gerard Salton Award.

Probably Irrelevant is a group blog on information retrieval and all things related. It serves as an open forum for IR research and development discussion. We aspire to have a wide range of IR researchers and practitioners contribute to the blog — from academia and industry, professors and students, evangelists and critics.

Of course, if you’d like to contribute, please leave a comment or contact us.