Editorial: To Benchmark, or Not to Benchmark?By Nilson Jacques
Vaguely interesting, you might say, but how does this affect my job day-to-day as a developer? Benchmarks are often cited when trying to convince people of the benefits of framework y over framework x, and some people put a lot of importance in these numbers. Last week I noticed a new UI library called MoonJS doing the rounds on some of the news aggregators. MoonJS positions itself as a ‘minimal, blazing fast’ library, and cites some benchmark figures to try to back that up.
There are other reasons to be cautious when comparing libraries based on their claimed speed. It’s important to remember that, like SunSpider, many benchmarks are microbenchmarks; They are measuring repeated operations on a scale that you’re unlikely to match when creating interfaces for your applications.
It’s also worth asking how important speed is for your particular use-case. Building a bread-and-butter CRUD app is unlikely to bring any UI library to its knees, and factors such as the learning curve, available talent pool, and developer happiness are also important considerations. I’ve seen many discussions in the past on whether Ruby was too slow for building web applications but, despite faster options existing, a good many apps have been and continue to be written in Ruby.
Speed metrics can be misleading, but they may also be of limited use depending on what you’re building. As with all rules of thumb and good practices, it’s always good to stop and think how (or if) it applies to your situation. I’m interested to hear your experiences: Have you used software that didn’t live up to its benchmark claims in practice? Have you built apps where that difference in speed was important? Leave me a comment and let me know!