Groovy, an Open Source Success Story

It’s Open Source Week at SitePoint! All week we’re publishing articles focused on everything Open Source, Free Software and Community, so keep checking the OSW tag for the latest updates.

Apache Groovy is a multi-faceted general purpose programming language for the Java platform. While primarily an object-oriented language with many dynamic language features, it also supports functional programming, static type checking and static compilation. This article looks at some interesting aspects of Groovy’s history and some of the significant guiding principles which help keep it a vibrant open source project.

What is Groovy and why should I care?

There are many available programming languages, and several popular alternatives to Java that are specifically designed to run on the JVM, so what’s so special about Groovy? Groovy is particularly known for its:

• close integration with and low learning-curve from Java
• multi-paradigm support for functional and object-oriented styles
• flexible syntax and productive libraries
• extensibility through metaprogramming
• scripting capabilities
• support for domain specific languages (DSLs)

It’s also one of the most widely used alternative languages for the JVM with downloads of about 12 million per year and steadily rising. But this isn’t an article to sell you on the features of Groovy, rather it describes the role open source played in Groovy’s evolution.

Groovy has been an open source project since its inception. It started life from the desire to be able to program using less boilerplate code like you could using Ruby, Python or Smalltalk but on the JVM platform and using a syntax that a Java programmer would find natural. Without that spark of inspiration, Groovy wouldn’t exist, but that inspiration alone didn’t bring Groovy to where it is today. A number of other factors were crucial, and various hurdles needed to be overcome. Let’s look at some of those factors – different pillars, if you like, supporting a successful open source project.

Building a better light bulb but standing on the shoulders of giants

A key motivator behind many of the early features of Groovy were things that the project members deemed were needed but missing in Java. However, the Groovy team was much smaller than the engineering team behind Java. So an underlying strategy was made to let Java do the things it does well and Groovy would focus on the gaps: Don’t do all things but do the things you do well.

The integration between the two would be kept closer than what other alternative JVM languages had attempted. When new versions of Java came out, sometimes with features inspired by what Groovy had added years earlier, Groovy would just try to move the goalposts out further with the next round of missing features.

Having said that, it is worth noting that Groovy’s addition of static type checking and static compilation to get better compile-time errors and dramatically improve performance does overlap somewhat with Java. Even though Groovy allows seamless interplay between mixed Java and Groovy codebases, many users just didn’t want to leave Groovy and have to revert back to Java yet wanted some of the benefits from Java’s static typing and type checking. Of course Groovy, always wanting to raise the bar further, made the type checker extensible so that you could add your own custom checkers for stricter-than-Java checking if you desire.

As well as building upon Java, many early Groovy features were inspired heavily by features found in Python, Ruby, Smalltalk and other languages – no need to reinvent the concept, just provide a version that is JVM friendly and intuitive to someone with a Java developer mindset. Examples of such features include:

• Groovy’s metaprogramming capabilities and range data type borrowed ideas heavily from Ruby
• the list and map literal notation and the syntax for default parameters were inspired by Python
• the collection processing methods, collect and inject, followed Smalltalk’s naming scheme
• closures came from the world of functional programming

In recent years, the nice observation is that Groovy features have found their way back into languages like C#, Swift, Scala, Kotlin, Frege and even Java. Some examples include:

• Java’s lambda expressions can be thought of as a limited version of Groovy’s closures
• as part of project Coin, Java added strings to switch statements (Groovy allows strings, regex expressions, class expressions and closures as possible targets for a case statement)
• the syntax for the Elvis operator ?: is now used in PHP, Fantom and Kotlin though they test against null rather than applying Groovy truth; numerous languages have instead added a ?? operator for null coalescing including C# and Swift
• the safe navigation operator ?. is now in C#, Swift, Kotlin, Fantom and Ruby (as &.)
• the spaceship operator <=> is now in PHP, Ceylon and Ruby
• numerous dynamic languages have looked at ways to add static typing checking

The fact that Groovy is open source and its design and implementation are out in the open make this exchange of ideas possible; fostering innovation and raising the bar for all open source languages.

Professional leading-edge development

A common characteristic of many (but not all) open source projects is a high-quality codebase and often the pioneering use of good development practices. People interested in best-practice techniques are also often the ones looking for the latest technology that might ease some of their pain points. Groovy has certainly been fortunate to have such individuals involved within its community at various points. Numerous techniques and approaches have been incorporated along the way to keep code quality high, improve test coverage, maintain appropriate compatibility and adopt best-of-breed devops practices to lower the barrier of entry for potential new contributors and minimize the workload for existing project members. Two examples to illustrate the point are:

• every PR that is submitted is tested automatically on a CI server; Groovy’s committers know if a submission would break tests without having to manually apply and run the tests themselves
• Checkstyle was an important additional check on the Java code sources but doesn’t work on Groovy code, so the Codenarc project was created to provide a similar checking framework for Groovy code

It might sound like the project has a heavy-handed set of policies and procedures that developers need to follow but that isn’t the case. Indeed, the project’s governance, i.e. how it is run, is very lightweight. Good practices are embedded within the project artifacts and practices are encouraged and reinforced as needed. As new contributors join the team, existing committers give guidance as to why things have been done a certain way in the past. We’ll come back to say a little more about governance shortly.

One of the commonly heard agile mantras is release early, release often. Certainly Java’s historically very slow release cycle and slow adoption of new features is one of the reasons for Groovy’s existence. Groovy has tried to improve upon that dramatically, by providing new features much earlier and at a much improved pace. Having said that, sometimes technical issues have even slowed Groovy releases, but that is being worked on. Of course, being a language, we don’t want to force our users to have to upgrade every week, so we don’t need to be as aggressive as some other projects.

Keeping it fun

While striving for professionalism and quality, the project team has also taken a stance of pragmatics over theoretical purity. Groovy users want to get things done and have fun doing it; the language shouldn’t get in the way. With this in mind, Groovy has adopted a somewhat agnostic position on how the language should be used. It supports object-oriented and functional styles, imperative and declarative. It strongly encourages quite a few good practices but doesn’t get in the way with the particular style a programmer may choose. It has a somewhat mature approach to how developers should be treated. Try not to let developers shoot themselves in the foot but don’t prohibit them from doing so if they really go out of their way to do it.

Another aspect of project governance that can dampen contributor enthusiasm is a heavyweight approval process. As mentioned before, the Groovy project tries to keep things as lightweight as possible. It doesn’t have an elaborate process for evolving the language like Java’s JCP process. Minor non-controversial improvements to the language, might be proposed, discussed and approved through consensus on the mailing list backed up by a proposed pull request on Github. For larger changes to the language, Groovy has a Groovy enhancement proposal (GEP) process to allow more elaborate considerations to take place. It’s not all plain sailing though. One of the hardest parts about running an open source project is often not about working out what new features you could possibly add but rather deciding what to leave out or resisting the urge to add a feature too early before its implications are fully understood.

When needed, Groovy does have a formal governance structure to fall back on, so we might, for instance, have a formal vote about whether a particular feature should be added or not. The project follows the typical Apache governance model having a project management committee (PMC) responsible for managing the project. So in summary, we strive for consensus within the broader community first, then the committers and finally, if needed, the PMC.

Finally, Groovy has maintained a friendly vibe throughout its community. There are times when there are heated debates and people agree to disagree but on the whole, participants within the community defer to the greater good of the community. All in all, things work reasonably well; Groovy users don’t want to argue endlessly about academic points of theory, they want to get stuff done while having fun. It’s the combination of these factors that has attracted individuals to start and continue contributing to Groovy or to other projects within the Groovy ecosystem.

Longevity beyond any one individual

Individuals contribute to open source projects for a variety of reasons. Some might be fully or partially sponsored by their companies, others might be volunteers who might just enjoy working with a world-wide community of like-minded individuals. Given the varying motivations of these contributors, an open source project has much less control over when any individual contributor may come or go as their individual circumstances change. Over the years, Groovy has seen some major individuals come and go and others whose circumstances have changed the way they can contribute – but the project has adapted when needed and continued to flourish.

Groovy has had over 300 code committers during its lifespan and has averaged over 50 active code contributors per year during the last 5 years – a higher number than at any previous time. For those interested, the top 100 contributors’ ten-year history is available online via Github graphs. Numerous other contributions happen outside code contributions too, including on mailing lists, forums and blogs to name just a few.

Probably the most controversial person to lose from a language is one of its founders and indeed a lot of publicity was raised when Groovy’s founder, James Strachan, a prolific open source contributor moved on from the project to focus his time on higher priority activities. James is famously quoted in 2009 as saying: “I can honestly say if someone had shown me the Programming in Scala book by Odersky et. al. back in 2003, I’d probably have never created Groovy”. Many in the Scala community have gone on to quote and misquote these words over the years. James went on to clarify that he never regretted creating Groovy and after diving deeply into Scala he altered his fondness of the language somewhat and moved on to other languages. In more recent tweets he has praised Groovy and has slides showcasing Groovy in some of his recent talks. He is particularly fond of Groovy’s DSL capabilities in contexts like Jenkins pipelines within a continuous delivery pipeline. Perhaps this tweet captures his current thinking:

I still love groovy (jenkins pipelines are so groovy!), java, go, typescript and kotlin. I don’t use scala any more

— James Strachan (@jstrachan) October 7, 2016

The peaks and troughs of sponsorship

Organizations sponsor open source development for a variety of reasons. It might give them a competitive advantage, improve security, foster innovation, motivate staff, improve profit margins and time to market. It’s easy to see why many organizations have some kind of involvement with open source. But more often than not, monetizing opportunities come from as high up the “software food chain” as possible.

Sometimes the link between revenues earned and a general purpose programming language like Groovy can involve several layers. Money might be made from an app or web service or cloud offering that is build using a framework that is written in Groovy, or built using Groovy, or tested with Groovy. It’s for this reason that we should applaud the investments from a number of organizations who have sponsored people to work on Groovy including G2One, Canoo, ASERT, SpringSource, VMWare, Pivotal, and OCI.

Probably the most well-known of these companies within the Java community in recent years is Pivotal. It was a spin-off from VMWare which earlier had acquired SpringSource. Groovy (and the Grails web framework) formed parts of its product portfolio. Very valuable contributions were made as part of this sponsored work and in many cases it was focused on features of particular value to users of Groovy within the sponsoring organization. But, to the credit of the companies involved, no heavy-handed influence was put on the overall governance of the language which allowed it to remain as a general purpose language of appeal to many people.

During the Pivotal sponsorship period, which lasted about 2 years, contributions continued from outside the sponsoring companies and interest in Groovy continued to rise. Later, when Pivotal’s directions changed and their investment in full-time contributors to Groovy (and the Grails web framework) ceased, the Groovy project again adapted and contributions from other sources become paramount again.

The easiest way to see the impact of Pivotal’s withdrawal is by looking at the numbers. During the approximately 2 years of Pivotal sponsorship there were 6 Pivotal code contributors and 111 others. During the 18 months since then, there have been 114 active contributors. This number includes some of the original Pivotal employees who now contribute under new roles in other organization or on a volunteer basis. So with a very superficial analysis, it might seem that the numbers haven’t changed much – in fact, they might be argued to have improved since the second time period being examined is shorter. But that wouldn’t be an accurate reflection of reality. Some of the 6 Pivotal code contributors were working nearly full time on Groovy (some were primarily working on Grails) whereas some of the 111 others may have submitted only a handful of commits for the whole 2-year period.

A slightly better approach to analysing contributions is to look at the number of commits. During the 2-year sponsorship period there was an average of 359 commits per quarter, 158 per quarter from Pivotal staff and 201 per quarter from others. Since then there have been 188 commits per quarter on average. So, there is a noticeable reduction in contributions since Pivotal’s sponsorship ceased, but the current level of commits remains healthy and the number is on the rise again.

We should point out that the number of commits is by no means a perfect metric, either. Indeed, a commit to fix a small typo clocks up the same weight as a complex commit adding a new feature like traits or static type checking – though neither of those was (or would be expected to be) added with a single commit. Perhaps the real worry about the Pivotal withdrawal is that complex features (which can best be tackled by teams) might be difficult to progress with a large number of contributors each having only a small amount of time. This is indeed something that needs to be monitored over time. All we can say is that the signs are encouraging. The number of contributors is increasing and some of the progress that is being made (like the new “Parrot” Antlr-4 based parser) surpasses progress made at any previous time.

There’s no place like home

In the early years of Groovy, it’s home was as part of Codehaus’s portfolio of open source hosted projects. Codehaus was instrumental in providing the necessary infrastructure for Groovy’s early days. It provided us with a full suite of best-of-breed tools: source control, mailing lists, issue tracking, downloads, websites/wiki, continuous builds, and chat, all before Github or cloud CI services such as Travis CI even existed. Having Codehaus take on the burden of hosting and maintaining all the necessary infrastructure for running a successful open source project allowed the project team to focus on just building the best Groovy they could. Codehaus were fairly good at bringing on board useful new technologies, so Groovy adopted distributed version control fairly early on which opened up a better workflow for evolving the codebase. It wasn’t long before the project also started using Github. Over and above the git repo that Codehaus provided, Github’s web-based workflow for cloning repos and creating pull requests significantly lowered the barrier of entry to potential new contributors while at the same time, minimised overheads for the project team when vetting, discussing and accepting new contributions.

At about the same time as Pivotal sponsorship was coming to an end, the Codehaus organization was winding down its open source hosting service. With the closure of Codehaus looming, Groovy needed to find an alternative “hosting service” for its infrastructure needs – just Github alone wasn’t perceived to provide everything we needed. While discussing options within the community, the issue of project governance was raised. Codehaus had a defacto governance that Groovy was operating under. The community wanted to be sure that wherever Groovy ended up, that not only the technical artifacts of the project (codebase, mailing archive, issue archive, etc.) survived the move safely but the governance of the project going forward remained friendly to our existing users and peer projects within the Groovy ecosystem.

It was after numerous deliberations with these thoughts in mind that the Apache Software Foundation became the leading contender for a destination for Groovy. It provided its own similar infrastructure to Codehaus, allowed us to maintain a workflow involving Github that our users were comfortable with and, most importantly, it provided a rock-solid governance model and a strong emphasis on community. Not only did we know numerous fellow developers from within the Apache community already but the degree it embraces community felt a good fit for Groovy. Much to our delight, the move to Apache has shone a spotlight on Groovy and user contributions and downloads have never been higher and similarly language rankings of usage have increased; we had found our new home.

The Groovy ecosystem and community

A discussion about Groovy would be incomplete without mentioning our other main class of users. Our peer projects within the Groovy ecosystem including but not limited to Grails, Spock, Ratpack, Gradle, GPars, Griffon and Jenkins. These projects, and many more like them, provide higher-level services or frameworks that build on top of Groovy. Many users of those projects end up using Groovy, some of them not even necessarily realizing that fact. Without the feedback and support of these projects, Groovy would not be the useful language that it is.

Finally, it would be remiss not to mention the many users of Groovy over the years, many of whom have provided feedback and driven many changes in the language. Groovy today is a vast improvement over what it once was. There are still numerous improvements that we’d like to make and a huge raft of features and ideas that are waiting in the wings. The contributors and community are what will help make those changes become a reality. Hopefully, you have enjoyed learning about some of the pillars which have supported Groovy’s evolution. Why not give the language a try or come visit us.

Frequently Asked Questions (FAQs) about Groovy Open Source