Yext's build system ICBM: An example of "worse is better"

about | archive

[ 2013-October-06 18:17 ]

A couple weeks ago Yext engineer Austin Chu presented their build system called ICBM. While the talk was clearly a recruiting event, it was still pretty interesting. ICBM was developed to make building Yext's Java servers go fast. When building a single server, ICBM only builds the classes that are used, rather than all source files the way Ant and Maven do. The interesting part is how it does this. To find all dependencies, you need to parse the source code. Unfortunately, parsing Java is pretty complicated. ICBM side-steps this problem by doing something crazy: it uses regular expressions. When I first heard this, I immediately thought this was a disgusting hack. However, after listening to the rest of the talk and thinking about it a bit, I've decided this is actually an elegant hack, and a perfect example of "worse is better" design. Parsing imports from Java source code using regular expressions is not "correct," and I can produce code where ICBM won't find the right dependencies. However, real code that doesn't work should be exceedingly rare. For Yext, any cases that don't work can be fixed, since they own the code it needs to build. As a result, ICBM avoids a ton of complexity, and is probably faster to boot.

The other interesting feature is that ICBM builds single jar executables. This means all code and dependencies are in a single file, which makes deploying and rolling back very easy. I'm always happy to see others adopting this approach, since I think it is the right way to deal with servers. We do this at Mitro, using a quick-and-dirty script I hacked up. Dropwizard advocates using "fat jars," built by Maven Shade. Go produces statically linked binaries for the same reasons.

ICBM is another re-implementation of Google's build system (the others I know of are Twitter's Pants, Facebook's Buck, and Evan Martin's Ninja). It may be worth a look if you are dealing with a lot of Java code. Unfortunately, we don't have that much code at Mitro yet, so it doesn't really solve any problems for us, and we'll stick with plain ant. Our biggest build-related issues are more around things like managing third-party dependencies, managing debug versus release configurations, and pulling in other resources like HTML, templates, and JavaScript that may need additional processing. I'm still looking for the "perfect" build system ...