Lessons from the Jikes RVM

about | archive

[ 2005-November-01 18:08 ]

In the Java world, Jikes is a name that means "fast compiler." I am not sure if it is still the case, but IBM's Java compiler is much faster than Sun's. IBM Research has reused this brand for an experimental Java virtual machine, called the Jikes Research Virtual Machine (RVM). It has been used for over one hundred academic publications, both inside and outside of IBM. The original Jikes RVM team have a paper about open-source research projects that is quite interesting. It discusses the history of the project, which ends up discussing some of the major flaws in the academic system. Here are my comments about a few choice quotes:

[Researchers have] different motivations than most open-source software contributors; they are primarily driven by the desire to produce publications and technical results, not production-quality software.

This is, in my opinion, a big problem. In computer science, papers are not useful unless the ideas are actually used in a real system. One of the best ways to do this is to build prototypes that are actually used, in order to validate the ideas. The goal of computer science research should be to solve real problems.

... the vast majority of researchers choose not to make their implementations publicly available. We suspect that the main issue is that the research community, as represented by program committees and tenure committees, does not explicitly value producing open software.

Again, the authors have hit the nail directly on the head, although in my experience it makes no difference if the software is open or not: Producing software has no value in the academic world. For example, I have done a fair amount of work using the ns2 network simulator which definitely exhibits the "thousands of marginal grad student hacks" problem. The issue is that ns2 is extremely useful infrastructure that enables network research. However, no one has any motivation to invest the massive amount of time that would be required to improve it.

Unfortunately, there is no easy answer for how to solve these issues. The current metric for evaluating a researcher is their publication list. This metric is fundamentally flawed but there is no clear alternative. Part of the solution would be to raise the bar of what is required for a paper to get published. Today, there are many conferences that have horribly low standards. These conferences are completely ignored by researchers, unless they happen to find a paper that is directly related to their current work using Google Scholar or a similar database. In my opinion, these conferences should just disappear, as their role can be adequately filled by posting technical reports on university web sites. The other part of this challenge is that the top conferences are being crushed by the high number of submissions. The community as a whole must raise their standards, and not submit substandard work to these venues. If these two things happened, many fewer papers would be published, but the average quality would increase. As a result, there would be likely be more emphasis on other research contributions, instead of exploiting the paper publication game.