Directions in Search over Social Media
In his keynote at the Search in Social Media workshop at CIKM, Andrew Tomkins suggested that there is plenty of room for academic IR research progress in social media. I happen to agree.
Community generated content has been all the rage for a few years: blogs, Wikipedia, online forums, twitter, Yahoo! Answers, and the list goes on. Many of these generate a large volume of archived data — some in the form of more or less polished documents, like a blog post or Wikipedia article; others, like twitter, are snippets of an often one-sided conversation and broadcast messages.
From the IR researcher’s perspective, is it worth studying these artifacts of “social media”? Is there something that distinguishes these from other document collections? If so, how can we leverage that distinction in our retrieval models? This post aims to answer a couple of these questions and hopefully bring up a few more.
First and foremost, we need to identify whether there is value in providing access to artifacts of social media. Some, like twitter, seem to be mostly ephemeral, only (generally) interesting in the moment and quickly fading from view. Even the twitter search engine advertises: “See what’s happening — right now” and the results (as far as I can tell) are only ranked chronologically.
Many other types of social media — some existing long before Web 2.0 was born — can be real treasure-troves of information. There exists an online forum, public mailing lists, newsgroup or message board for virtually every special interest group under the sun — from gardening, to home-brewing, to apple computers. These are often heavily trafficked, populated with real subject matter experts, and host a rich information exchange. I would argue that the content created through these social media outlets present an enormous value to searchers, and information retrieval research has a lot to contribute in this corner of social media.
What makes these document collections different than what has been previously studied? Can we just treat them the same as web pages? Or do they need special consideration?
In many of these collections, the unit of retrieval — what we consider a document — is not fixed, but rather dependent on the task. Consider online forums, often organized into topical sub-forums, which in turn are organized into conversation threads of individual posts. Some information needs many only require a single post as a result, some require the context of the full conversation thread, and others may need to retrieve a pertinent sub-forum.
These collections often offer another orthogonal axis of retrieval — the author. In highly trafficked message boards and mailing lists, tens or hundreds of thousands of users with varying levels of expertise contribute to the conversation. One may wish to find subject matter experts to address a question to, or favor message threads with contributions from those more likely to know the answer.
These factors, of course, are not entirely unique to social media search, and have to some degree been addressed in previous research. This question of identifying the granularity of the unit of retrieval has been addressed at the document level (for example in XML element retrieval at INEX), but not so much at the collection level. Resource ranking in federated search and cluster-based retrieval bear some resemblance to the selection of a topical sub-collection, such as a sub-forum ranking. Author-ranking has also been studied at TREC in the Blog and Enterprise Tracks. But, each of these have been studied in isolation, without much regard to the interaction between the different aspects of the collection. To my knowledge, no IR testbeds exist that contain the rich collection structure offered in these types of social media.
This, in my mind, is the real promise of research in search over social media. These collections provide multiple levels of organizational granularity, different axes of organization, multiple types of searchable objects, and relations among those objects. I predict that this will be an interesting and fertile direction of information retrieval research — pushing the systems to support more sophisticated multi-dimensional indexing and extending existing retrieval models to handle rich relationships between documents.
Nice post!
The “orthogonal axes of retrieval” discussed here bring to mind faceted retrieval, which has been discussed for quite a while, but suffers from lack of collections to allow rigorous evaluation — but this issue probably deserves its own post :)
Interestingly, the research on book retrieval tries to address similar problems to the ones addressed here: multiple levels of granularity and organization of results by authorship, topic and date. For example, this poster: http://www2007.org/posters/poster901.pdf
A few things:
0. Please define social media. Does it have to do with the ratio of users generating content to those querying content? Or something else?
1. There are existing forum/mailing list/newgroup search engines. Do we know anything about how users interact with them?
2. You seem to be focusing on forum/mailing list/newgroup which all have the same discussion/qa/threaded format. Now what’s the relationship to other social media? What about these other media make them “uninteresting”? Is there a formal way to predict this?
Nice post.
Michael — thanks for the reference. Its certainly pertinent to this discussion, and an interesting extension to the types of research I associate with with INEX & XML element retrieval.
I agree, many of these collections to which I’m referring could lend themselves to faceted browsing interfaces, but I don’t think those types of interfaces “solve” the problems of information access. They provide users with control over how to display the data along different (orthogonal?) dimensions, but when there’s 200k different authors, does providing filtering or sorting on that attribute really help? I would argue that for faceted interfaces to be successful on large data sets such as these, we still need automatic methods for ranking authors, topics, threads, etc.
Fernando —
0: I’m partial to Marc Smith’s definition of social media: “collective good produced through computer-mediated collective action”. The volume of producers or consumers does not define what is social media, but certainly are useful dimensions to characterize social media. The “collective good” that I think is the most interesting from the IR perspective are long-lived text artifacts. Friend networks, “tweets”, link collections, tags all certainly have value, but are (IMO) less interesting from the IR research perspective.
1: I don’t know, but am working on it. I’ve been in contact with the people at BoardTracker, and have been promised some interaction data. I would love to get query logs form a service like MarkMail, but haven’t yet pursued it.
I have looked at the AOL query log data a bit, and there are a few message boards that receive 1-2k clicks over the three months of that collection. This gives a rough idea of kinds of queries that may be served by data in online message boards, but not a very rich picture of the interaction.
2: See (0) above. The archived Q/A dialog is what I’ve spent the most time looking at, but certainly not the only worthwhile social media to study from the IR perspective. Wikipedia comes to mind, where edit history provides another dimension of the collection structure to work with.
I’m attracted to online forums and mailing lists because they have a history of hosting an exchange with experts. I also think existing tools to search them are lacking.
0: I apologize for dwelling on definitions but I’d rather avoid a “you’ll know it when you see it” approach.
So when I hear,
“collective good produced through computer-mediated collective action”
This is how I interpret (text) collective goods and actions,
collective good: a corpus accessible to and of value to everyone
collective action: a corpus modifiable by anyone
Is this accurate?
2: Okay. I’m really hoping that if two corpora are referred to as “social media”, then they will share interesting retrieval techniques more so than a “social media” corpus and a news corpus.
0: no apologies needed! I think your interpretation is accurate, modulo varying definitions of “anyone” and “everyone” :)
2: I only partially agree on this point. As you seem to be implying, “social media”, especially the definition I cited, is overly-broad. Maybe intentionally so. I like this high-recall (low-precision?) definition, as it prompts us to re-evaluate how we think about existing collections that may or may not generally be considered “social”.
I do think current collections that are generally considered social media share common organizational patterns: Individuals typically have a persistent identity that is tied to their contributions to the collection. This allows us to treat authors as a unit of retrieval, for example. Objects in these collections often support the attachment of commentary by the community — tags, discussions threads, blog comments.
These organizational patterns may not be either necessary or sufficient to define a media as social. But, I think collections that share these types of organizational idioms should benefit from the same IR techniques that leverage of them.