JUnit 5 State Of The Union

JUnit 5 has been under development for about 14 months now and the prototype is almost a year old. Time to summarize what happened so far, where the project stands, and where it’s going.

All technical details are based on the current JUnit version, milestone 2. You will find more information in the official user guide, with separate versions for milestone 2 and the most recent build, or on my blog, where I wrote a series about JUnit 5, that I update whenever a new milestone is released.

Previously On Battlestar JUnit

Why JUnit 5? Why not 4.13, which has been in the making since, err, forever? Two and a half reasons…

Goodbye, JUnit 4

Part of JUnit’s success comes from its great tool support, for which tool creators went as far as using reflection (down to private fields) to access information that the API would not hand out. This bound tools to implementation details, which in turn made it hard for JUnit’s maintainers to evolve it – nobody likes breaking downstream projects. This is an important reason for why JUnit’s progress has basically come to a halt.

Then there are the runners and the rules. They were created to extend JUnit and did a good job at that – good but not superb. Runners are a heavyweight concept where one has to manage the whole test lifecycle. But worse, you can only use a single runner at a time – a very strict limitation. That’s why rules were introduced in 4.7. While much simpler and mostly composable, they have other drawbacks – namely that they are limited to certain kinds of behavior, which can best be summarized as before/after.

Last and least was the Java version. Everybody wants to play with lambdas nowadays and having JUnit 4 require Java 8 would have been a hard sell.

Hello, JUnit 5

So in 2015 a team formed around the idea of a complete rewrite. At first dubbed JUnit Lambda, the project did a successful crowdfunding campaign and came up with enough money and employer sponsorship to work on it full time for six months. Workshops, a prototype, an alpha version, and two milestones followed and since earlier this year the project is called JUnit 5. By now, the funds are used up and it is developed like so many other great open source projects: by dedicated people in their free time during the day’s first and last hours.

The team is currently completing the third milestone and has plan for a least two more before a release candidate is even considered. But we’ll come back to that. Let’s first look at what we’ve got in our hands at the moment.

What Does It Look Like?

Instead of going into details about the new API I ask you to imagine a full blown JUnit 4 tests with all the shenanigans the API offers. Here’s how JUnit 5 is different in comparison to that:

• Test classes and methods can be package visible.
• Interface default methods can be test methods.
• @Test no longer takes parameters. Expected exceptions and timeouts are realized via assertions.
• You can create tests dynamically at run time with @TestFactory annotated methods (more on that later).
• The lifecycle annotations got new names: @Before became @BeforeEach and @BeforeClass is now @BeforeAll; likewise for @After....
• @Ignore changed to @Disabled.
• In assertions, failure messages come last and can be created lazily.
• A new assertion assertAll executes the given checks and always reports all results, even if some assertions failed.
• @Nested can be used to run tests in inner classes.
• @DisplayName assigns human readable names to tests.
• Runners and rules are gone and replaced by a new extension mechanism that integrates seamlessly with the native API (more on that later).
• Tests can have parameters, where instances are created by extensions (again, later).

All in all, this is what a test class might look like:

@DisplayName("Some example test cases")
class ExampleTests {

    @BeforeAll
    static void initAll() { }

    @BeforeEach
    void init() { }

    @Test
    @DisplayName("Custom test name containing spaces")
    void succeedingTest() { }

    @Test
    void groupedAssertions() {
        // In a grouped assertion all assertions are executed, and any
        // failures will be reported together.
        assertAll("address",
            () -> assertEquals("John", address.getFirstName()),
            () -> assertEquals("User", address.getLastName())
        );
    }

    @Test
    @DisplayName("😱")
    @Disabled("for demonstration purposes")
    void skippedTest() {
        // not executed
    }

    @Test
    @CoolInstanceProvider
    void testWithParameters(MyCoolClass coolInstance) {
        // do something with 'coolInstance'
    }

    @TestFactory
    Collection<DynamicTest> dynamicTestsFromCollection() {
        return Arrays.asList(
            dynamicTest("1st dynamic test", () -> assertTrue(true)),
            dynamicTest("2nd dynamic test", () -> assertEquals(4, 2 * 2))
        );
    }

    @AfterEach
    void tearDown() { }

    @AfterAll
    static void tearDownAll() { }

    @DisplayName("Some nested test cases")
    class NestedTest {

        @Test
        @DisplayName("╯°□°）╯")
        void failingTest() {
            fail("a failing test");
        }

    }
}

As for @Nested and @DisplayName, this is what it looks like to run an example written by the JUnit team in IntelliJ 2016.2:

Not bad, right? For more have a look at the user guide or my post covering the basics.

How Can I Use It?

To start writing tests all you have to do is include the API artifact org.junit.jupiter:junit-jupiter-api:5.0.0-M2 in your project. Unfortunately, running tests is a little more complicated…

IDEs

Native IDE support is a must-have for any serious testing framework. JUnit 5 is not there yet but that’s understandable considering that it is not even officially released.

• IntelliJ, since 2016.2, has built-in support that works reasonably well.
• The Eclipse team started working on it – see the corresponding issues in JUnit’s and in Eclipse’s bug tracker.
• NetBeans has no support and I could not even find an issue.

Build Tools

What about native build tool support? The JUnit team implemented a Maven Surefire provider and a Gradle plugin but these are just proofs of concept. To fully integrate and battle-test them, the team would like the respective projects to take over these code bases.

• For Maven that process just started by relicensing the source code under the Apache License. An issue in Apache’s JIRA tracks the progress.
• I saw no Gradle-related activity on the JUnit project and could not find an issue related to JUnit 5.

Fallbacks

This situation is a little unsatisfactory but luckily there are alternatives. One is to simply wrap all JUnit 5 tests in a JUnit 4 test suite and rely on JUnit 4 support:

@RunWith(JUnitPlatform.class)
@SelectPackages({ "my.test.package" })
public class JUnit5TestSuite { }

See the user guide for setup details.

The other alternative is the console launcher. Again, see the user guide to get started and check out my JUnit setup post if you run into problems. When you’re done, you can run tests as follows:

junit-platform-console -p -p target/classes/:target/test-classes -a

Outer Space

Now that you know how to write and run JUnit 5 tests, let’s go beyond the basics and take a peek into more advances topics.

Dynamic Tests

Dynamic tests are a new concept in JUnit 5. They allow the easy creation of tests at run time that tools will recognize as full test cases.

An example explains this best. Assume you have a method testPoint(Point p) that asserts something on the given point and you want to run that test for a host of points. You could write a simple loop around it:

@Test
void testPoints() {
    getTestPoints().forEach(this::testPoint);
}

This has the substantial drawback that while, conceptually, these are many tests, to JUnit it’s just one. First among the undesired consequences is that as soon as the first test fails all others aren’t even executed. Another is that tools, IDEs in particular, will report this as a single test, which is not ideal either.

JUnit 5 gives us an out. Test code can be wrapped into a DynamicTest instance that is created at run time. Methods that return them can be annotated with the new @TestFactory annotation and JUnit will call them, insert the created tests into the test plan, and execute them.

@TestFactory
Stream<DynamicText> testingPoints() {
    return getTestPoints()
        .stream()
        .map(p -> dynamicTest("Testing " + p, () -> pointTest(p)));
}

Neat. And if you really want to, you can use dynamic tests to write tests with lambda expressions

Extension Model

I already alluded to JUnit 4’s runners and rules. They are gone, replaced by extension points, which third party code can interact with. Here’s how that works.

Extension Points

For most steps in a test’s lifecycle an extension point exists:

• Test Instance Post Processor
• BeforeAll Callback
• Test and Container Execution Condition
• BeforeEach Callback
• Parameter Resolution
• Before Test Execution
• After Test Execution
• Exception Handling
• AfterEach Callback
• AfterAll Callback

For each of these extension points exists an interface and since the points are very focused the interfaces are quite simple. Here’s the one for the Before Each callback:

public interface BeforeEachCallback extends Extension {

    void beforeEach(TestExtensionContext context) throws Exception;

}

I already bullet-pointed out that tests can have parameters. Since JUnit has no idea how to create an instance of MyCoolClass, extensions have to provide them. Here’s that extension point:

public interface ParameterResolver extends Extension {

    boolean supports(
            ParameterContext parameterContext, ExtensionContext extensionContext)
            throws ParameterResolutionException;

    Object resolve(
            ParameterContext parameterContext, ExtensionContext extensionContext)
            throws ParameterResolutionException;

}

For each parameter, JUnit will call support on every registered extension, throw an exception if not exactly one of them returns true, and otherwise request the one extension to resolve the parameter.

As a proof of concept, the team wrote a Mockito extension, which mocks parameters marked with @Mock. It can be used as follows:

void myCoolClassTest(@Mock MyCoolClass instance) {
    // ...
}

Writing Extensions

To create an extension you simply implement the interfaces corresponding to the extension points you want to interact with and register your class with JUnit. During test execution JUnit will pause at each extension point, gather all relevant information into an ExtensionContext instance, and call all extensions that registered with that point.

The extension context gives access to many interesting things, for example the test class or method as AnnotatedElements, so an extension can reflect over them. Then there is the ExtensionStore. Because JUnit has no interest in tracking extension instances, it makes no guarantees about their lifecycle. Extension classes should hence be stateless and can use the store to persist information. Another interesting aspect is the TestReporter, which allows extensions (and tests for that matter) to publish messages in a way that can easily be picked up by external tools.

Put together, a benchmark extension that simply reports a test’s run time looks as follows:

class SimpleBenchmarkExtension
        implements BeforeTestExecutionCallback, AfterTestExecutionCallback {

    // we need a namespace to access the store and a key to persist information
    private static final Namespace NAMESPACE = Namespace
            .create("com", "sitepoint", "SimpleBenchmarkExtension");
    private static final String LAUNCH_TIME_KEY = "LaunchTime";

    @Override
    public void beforeTestExecution(TestExtensionContext context) {
        storeNowAsLaunchTime(context);
    }

    @Override
    public void afterTestExecution(TestExtensionContext context) {
        long launchTime = loadLaunchTime(context);
        long elapsedTime = currentTimeMillis() - launchTime;
        report(context, elapsedTime);
    }

    private static void storeNowAsLaunchTime(ExtensionContext context) {
        context.getStore(NAMESPACE).put(LAUNCH_TIME_KEY, currentTimeMillis());
    }

    private static long loadLaunchTime(ExtensionContext context) {
        return context.getStore(NAMESPACE).get(LAUNCH_TIME_KEY, long.class);
    }

    private static void report(ExtensionContext context, long elapsedTime) {
        String message = String
                .format("Test '%s' took %d ms.", context.getDisplayName(), elapsedTime);
        // 'publishReportEntry' pipes key-value pairs into the test reporter
        context.publishReportEntry(singletonMap("Benchmark", message));
    }

}

Registering Extensions

I glossed over how extensions can be registered. But a very cool feature is hiding here! Read my post on extensions to find out how you can get your extension to look as follows:

@Benchmark
@Test
void test() {
    // ...
}

Architecture

Quick dive into the architecture, which is really interesting. To separate concerns of testers, extension developers, and tools as well as to allow other testing frameworks to benefit from JUnit’s great tool support (well, it’s not quite there yet but it will be) JUnit 5 is split into three subprojects:

• JUnit Jupiter provides the API that we have talked about so far and an engine that discovers and runs tests.
• JUnit Vintage adapts JUnit 3/4 to run within JUnit 5’s new architecture, which means it also provides an engine.
• JUnit Platform is where the magic happens. It provides an API for tools to instruct test execution and when that gets called, it locates test engines (like the two above), forwards the instruction to them, and lets them do their thing.

For one this makes it easy to run JUnit 3, 4, and 5 tests within the same mechanism. But the separation between the platform, which contains the engine API, and the engine implementations also allows the integration of all kinds of testing frameworks into the JUnit mechanism. Assuming a TestNG engine would exist, this is how that would turn out:

This is a big deal! Creating a new testing framework is an uphill battle because adoption depends on tool support and tool support depends on adoption. Now frameworks can break out of that vicious cycle! They can implement a JUnit engine and get full support in many tools from day 1.

This might give rise to a new generation of testing frameworks!

The Road From Here

JUnit is in very good shape and can already be used to replace vanilla JUnit 4. But, in accordance with the Pareto Principle, there is still some work to be done before it is ready to fulfill all use cases.

Ongoing Discussions

A lot of very interesting topics are being discussed!

JUnit 4 Compatibility

Wouldn’t it be great to create adapters for existing JUnit 4 rules? That would make migrating tests to JUnit Jupiter’s API (as opposed to just running them within JUnit 5) a lot easier. Issue #433 is exploring exactly that. But code is not the only good thing a project can produce. Issues #169 and #292 aim at providing a good migration guide from 4 to 5.

Extension Model

Personally, I think something’s missing from dynamic tests. It would be great for extensions to interact with this feature! One way to do that would be an extension point into which third party libraries could hook to inject dynamically created tests. They could then come up with interesting APIs to parameterize tests, for example. Talk is cheap. though, so I created an issue and a prototypical implementation for that feature. I hope it will be looked into during development of milestone 4.

Another detail is that while test factory methods are part of the full lifecycle, the dynamic tests they create are not. Meaning before and after callbacks or parameter resolution do not work for dynamic tests. This is tracked by an issue and I am sure a solution will be found before the release.

It is also currently not possible to interact with Jupiter’s threading model. This is a problem for extensions that want to push tests onto other threads, for example to make sure that Swing or JavaFX tests run on the Event Dispatch Thread or the Application Thread, respectively. There are several issues addressing this: #16, #20, #157

Other interesting topics:

• extension points…
  • to edit the test plan after discovery, #276
  • to alter test results, #542
• a global extensions registry, #448
• extensions on fields, #497
• an API for programmatic extension management, #506

Scenario Tests

A big topic are scenario tests, which is JUnit’s nomenclature for tests that are ordered and operate on common state (tracked by #48). TestNG offers this already and is an important reason for people to move away from JUnit. Well was a reason.

Let’s say we have a class encapsulating a REST API. With scenario tests we could have a first test that creates an instance, a second test could initialize it, a third logs in a user, and so forth. The state remains from test to test and those are mutating it in a fashion that prepares it for the next.

Others

Here are a couple of other interesting ideas:

• a mechanism for the launcher to detect engine capabilities, #90
• sanity checks for test code written by developers, #121
• lots of improvements on the console launcher, search filter
• a more flexible threading model, e.g. to parallelize tests, search filter

More Milestones

So what’s the plan for the following months? Sometime during the next couple of weeks milestone 3 will be released, which focuses on interoperability with JUnit 4 and an improved API for tools. After that, two more milestones are scheduled, number 4 focusing on parameterized tests, enhanced dynamic tests, and documentation, and number 5 on scenario tests, repeated tests, and test execution in user-defined thread.

Summary

We took a quick tour through the API we’re going to write tests against and saw that it was similar to JUnit 4’s but contains a lot of thoughtful additions. Most notably are built-in support for nested tests, human-readable display name and dynamically created tests. The extension model is very promising and the architecture, splitting JUnit 5 into Jupiter, Vintage and Platform, is just gorgeous, endowing other test frameworks with full tool support as soon as they’re implementing an engine.

On the soft side, we’ve gone through the history behind JUnit 5 and spent some time discussing the current state of tool support, which – with the exception of IntelliJ – is still lacking. We finally looked at some of the ongoing discussion and future developments, which I think are as thrilling as they are promising. I can’t wait to see the JUnit 5 GA release – hopefully some time in 2017.

Frequently Asked Questions (FAQs) about JUnit 5