Software Development? We’re Doing it Wrong.

“You are wasting an outlandish amount of money writing code that already exists”

Joel Spolsky, Fogcreek, StackOverflow

In software development, we like to throw acronyms around, such as DRY, KISS and OAOO, which stand for supposedly better practices that lead to a more effective development process. But is the process used to develop software today really efficient?

Code reuse is largely accepted as “best practice” in our industry. Code reuse has many advantages, such as a smaller code-base which is easier to grok for developers. It’s easier to maintain and optimize, since you can make less changes in order to achieve the desired results. Many modern techniques and tools were created in order to better reuse and modularize code.

But while we’re reusing code, we’re duplicating effort. The same code is being written over and over in different places, wasting time and money.

As much as we’d like to think it, our industry is not yet mature.

Custom development should be the last resort

When you look at more mature industries, such as the auto industry, no-one makes their own screws, or their own wheels, pipes and so forth – they buy off-the-shelf components, which are mass manufactured by companies that know how to make those products best. This leads to both better products – through years of field testing and experience – and to much lower costs.

Do you think the auto industry would be profitable if they produced every (or any) component from scratch and by hand? So how come we have this very process in the software industry, only we call it “custom development”?

As software developers, we pride ourselves on our skills. We want to develop stuff from scratch, because that is often the interesting part. But – every piece of code that solves a problem that had already been solved, is wasted effort.

For our industry to become mature and take the next step in both innovation and cost effectiveness, we need to rethink some of the processes we use today.

The future of the industry is component based development

Enter open-source

Open-source is one of the greatest innovations our industry has experienced in the short time it’s been around. Open-source offers standard source-code solutions for many common problems, and has revolutionized the industry with products such as Linux, MySQL, Apache and others.

Free, open-source libraries and frameworks exist for almost every major programming language and platform. The amount of code that is available is huge and benefits the software on many levels: aside from significantly reducing the time needed to build many products, developers are basically sharing their experience and knowledge through the availability of their code. Becoming a programmer today has become much easier in many ways because of open-source.

Still a huge divide between open-source and commercial software

So why isn’t every common need met by an available open-source solution?

There are a few reasons for that:

• Most open-source projects are created on developers’ free time and good will – developers who have day jobs or work freelance to make a living. Unless they catch major traction, they remain side-projects that are often abandoned gradually until they are not supported in practice. Only 7% of open-source projects are actively maintained.
• Anyone can publish an open-source project – there is no quality assurance or guarantee that it’ll meet certain standards. Many amateur developers get started by publishing open-source projects before they can produce a solution that could become an industry standard.
• Discovery is a problem – due to the long tail of abandoned projects and varying quality, unless the project is visible and well known, many developers will prefer to develop a solution themselves.
• As widespread as it is, open-source still covers a relatively small proportion of common needs in software development. There are so many more needs than solutions that custom development is often the only option.

If all the code written for commercial products was repackaged and released as small open-source projects, then we would have already a solution for most common needs.

That is unlikely to happen though – the sharing incentive just isn’t strong enough for most people. It’s similar to the participation in generated content sites such as wikipedia – only 1% of the audience is actively creating while the others merely consume.

An evolved model – Commercial open-source

So where do we see the industry going in order to break through and become more standardized and streamlined? The answer, in my opinion, is provided by products such as MySQL, RedHat and Magento, which use a business model we like to call commercial open-source.

Those products successfully built a commercial model around open-source, by either dual-licensing (free version + premium version), offering paid support and updates, or variations thereof.

I believe that building a viable business case around the release of source-code projects is the only repeatable and sustainable model for most open-source projects, and this is the direction in which the industry will move in the next couple of years. As releasing code become a viable business, several positive things will happen:

• Open-source projects will be much more actively maintained. If you can support yourself by supporting your own personal projects instead of doing contract client work or working as a hired man in a company, I think it’s a no-brainer.
• Much more code will be released into the market. This is a very important point – in my opinion, the amount of available, ready-to-use solutions will grow by a factor of 10. So, while individual solutions will cost more than free, the overall cost reduction to software projects – compared to the cost that would be spent custom-developing those solutions – would be enormous.
• Once you turn the maintenance of open-source into a business, expectations also rise. I expect increased overall quality as well as curation services that help determine which projects have a quality code-base as a reference for developers.

An evolution is occurring, whether you like it or not

I have witnessed what I’ve written about here many times over years as a software developer and the founder of a development firm.

It is what I learned that spurred me to co-found Binpress, a discovery service and marketplace for source-code. I believe this is how relevant source-code solutions will be found in the future.

By curating the code and adding a supporting commercial layer around it, we are trying to evangelize this approach and help make it a reality.

I realize that there is a strong ideological resistance to this kind of movement. Open-source was originally created according to an ideal – which I can only marvel at and applaud, having been so successful in an ecosystem that is mostly motivated by a capitalistic approach.

I do think, though, that as an ideal, we’ve seen the best that open-source can do. The acknowledged successes will be very hard to replicate.

What we have currently is increasing noise, as more and more people have easier access to start programming and publishing their projects. We need to streamline the process and add that missing financial element that makes it sustainable and repeatable.

I’m sure you have your own opinion to offer on this subject, and would love to discuss it in the comments.