Is the science of IR improving?
I’m just back from the annual meeting of ASIST (American Society for Info Science and Technology) in Columbus, OH. I gave a talk during one of the five sessions on IR, and after all the speakers were through there was a session of audience questions. Andrew Dillon lobbed a provocative question our way: how do we know if IR as a field is making forward progress? (I’m paraphrasing, of course). An uncomfortable pause set in, followed by obligatory sidestepping, e.g. “first we need to define progress.” It’s a fair question, though: we see incremental progress reported in the literature, but getting a high-level sense of the field’s forward motion strikes me as harder to come by.
I offered an off-the-cuff answer that I suspect readers might comment on. Actually it was two answers.
First, surely there is meaning in the increasing competition to publish in the field’s best venues. This isn’t news, but the following figure showcases the fact that getting a paper into SIGIR is indeed growing more difficult (many more people are trying).
Of course SIGIR is not synonymous with the field, but I think the figure speaks to the question Andrew asked. Unless the SIGIR community is spinning its wheels, increasing competition among researchers suggests expectations and standards for “successful research” is climbing.
My second answer had to do with the diversity of tasks that fit under the umbrella term of IR. Looking at TREC over the years we see new tasks appear (and disappear), new problems to tackle. I argued that the field is indeed making progress, and we can see that progress in this creativity. We are solving problems that we didn’t know existed (e.g. adversarial IR) or that actually didn’t exist (e.g. blog search) only several years ago. Does this creativity imply improvement? I argued that it does.
