JavaScript
Article
By Jani Hartikainen

JavaScript Testing Tool Showdown: Sinon.js vs testdouble.js

By Jani Hartikainen

Two Greek warriors with shields battle bugs

When unit testing real-world code, there are many situations that make tests hard to write. How do you check if a function was called? How do you test an Ajax call? Or code using setTimeout? That’s when you use test doubles — replacement code that makes hard to test things easy to test.

For many years, Sinon.js has been the de-facto standard in JavaScript tests for creating test doubles. It’s a must-have tool for any JavaScript developer writing tests, as without it, writing tests for real applications would be nigh impossible.

Recently, a new library, aptly named testdouble.js, has been making waves. It boasts a similar feature set as Sinon.js, with a few differences here and there.

In this article, we’ll look into what both Sinon.js and testdouble.js offer, and compare their respective pros and cons. Will Sinon.js remain the superior choice, or will the challenger take the prize?

Note: If you’re unfamiliar with test doubles, I recommend reading my Sinon.js tutorial first. It will help you better understand the concepts we’ll be talking about here.

Terminology Used in This Article

To ensure it’s easy to understand what is being discussed, here’s a quick overview of the terminology used. These are the definitions for Sinon.js, and they can be slightly different elsewhere.

  • A test double is a replacement for a function used during a test. It can refer to any of the three types mentioned below.
  • A spy is a test double which allows the checking of effects without affecting the behavior of the target function.
  • A stub is a test double which replaces the target function’s behavior with something else, such as returning a value.
  • A mock is a different approach to stubs. Mocks contain built-in verification and can be used instead of a separate assertion.

It should be noted that one of the goals of testdouble.js is to reduce the confusion between this type of terminology.

Sinon.js and testdouble.js at a Glance

Let’s begin with a look at how Sinon.js and testdouble.js compare in basic usage.

Sinon has three separate concepts for test doubles: Spies, stubs and mocks. The idea is that each represents a different usage scenario. This makes the library more familiar to those coming from other languages or who have read books using the same terminology, such as xUnit Test Patterns. But the other side is that these three concepts can also make Sinon more difficult to understand when first using it.

Here is a basic example of Sinon usage:

//Here's how we can see a function call's parameters:
var spy = sinon.spy(Math, 'abs');

Math.abs(-10);

console.log(spy.firstCall.args); //output: [ -10 ]
spy.restore();

//Here's how we can control what a function does:
var stub = sinon.stub(document, 'createElement');
stub.returns('not an html element');

var x = document.createElement('div');

console.log(x); //output: 'not an html element'
stub.restore();

In contrast, testdouble.js opts for an API which is more straightforward. Instead of using concepts like spies or stubs, it uses language much more familiar to JavaScript developers, such as td.function, td.object and td.replace. This makes testdouble potentially easier to pick up, and better suited to certain tasks. But on the other hand, some more advanced uses may not be possible at all (which is sometimes intentional).

Here’s what testdouble.js looks in use:

//Here's how we can see a function call's parameters:
var abs = td.replace(Math, 'abs');

Math.abs(-10);

var explanation = td.explain(abs);
console.log(explanation.calls[0].args); //output: [ -10 ]

//Here's how we can control what a function does:
var createElement = td.replace(document, 'createElement');
td.when(createElement(td.matchers.anything())).thenReturn('not an html element');

var x = document.createElement('div');
console.log(x); //output: 'not an html element'

//testdouble resets all testdoubles with one call, no need for separate cleanup
td.reset();

The language used by testdouble is more straightforward. We “replace” a function instead of “stubbing” it. We ask testdouble to “explain” a function to get information from it. Other than this, so far it’s fairly similar to Sinon.

This also extends to creating “anonymous” test doubles:

var x = sinon.stub();

vs.

var x = td.function();

Sinon’s spies and stubs have properties which offer more information about them. For example, Sinon provides properties such as stub.callCount, and stub.args. In testdouble’s case, we get this information from td.explain:

//we can give a name to our test doubles as well
var x = td.function('hello');

x('foo', 'bar');

td.explain(x);
console.log(x);
/* Output:
{
  name: 'hello',
  callCount: 1,
  calls: [ { args: ['foo', 'bar'], context: undefined } ],
  description: 'This test double `hello` has 0 stubbings and 1 invocations.\n\nInvocations:\n  - called with `("foo", "bar")`.',
  isTestDouble: true
}
*/

One of the bigger differences relates to how you set up your stubs and verifications. With Sinon, you chain commands after a stub, and use an assertion to verify the result. testdouble.js simply has you show it how you want the function to be called — or how to “rehearse” the function call.

var x = sinon.stub();
x.withArgs('hello', 'world').returns(true);

var y = sinon.stub();
sinon.assert.calledWith(y, 'foo', 'bar');

vs.

var x = td.function();
td.when(x('hello', 'world')).thenReturn(true);

var y = td.function();
td.verify(y('foo', 'bar'));

This can make testdouble’s API easier to understand, since you don’t need to know what operations you can chain and when.

--ADVERTISEMENT--

Comparing Common Testing Tasks in More Detail

On a high level both libraries are reasonably similar. But what about common testing tasks that you might need to do in a real project? Let’s take a look at a few cases where the differences start to show.

testdouble.js has no spies

The first thing to note is testdouble.js has no concept of a “spy”. While Sinon.js allows us to replace a function call so that we get information from it, while keeping the function’s default behavior, this it not possible at all with testdouble.js. When you replace a function with testdouble, it always loses its default behavior.

This is not necessarily a problem however. The most common usage for spies would be using them to verify callbacks were called, which is easily doable with td.function:

var spy = sinon.spy();

myAsyncFunction(spy);

sinon.assert.calledOnce(spy);

vs.

var spy = td.function();

myAsyncFunction(spy);

td.verify(spy());

While not a big issue, it’s still good to know this difference exists between the two, as otherwise you may be surprised if you expect to be able to use spies in some more specific manner with testdouble.js.

testdouble.js requires more precise inputs

The second difference you’ll run into is testdouble is stricter about inputs.

Both Sinon’s stubs and assertions allow you to be imprecise about what parameters are given. This is easiest illustrated by an example:

var stub = sinon.stub();
stub.withArgs('hello').returns('foo');

console.log(stub('hello', 'world')); //output: 'foo'

sinon.assert.calledWith(stub, 'hello'); //no error

vs.

var stub = td.function();
td.when(stub('hello')).thenReturn('foo');

console.log(stub('hello', 'world')); //output: undefined

td.verify(stub('hello')); //throws error!

By default, Sinon doesn’t care how many extra parameters are given to a function. While it provides functions such as sinon.assert.calledWithExactly, those are not suggested as the default in the documentation. Functions like stub.withArgs also do not come with an “exactly” variant.

testdouble.js on the other hand defaults to requiring the exact parameters specified. This is by design. The idea is that if a function is given some other parameters unspecified in the test, it’s potentially a bug, and should fail the test.

It is possible to allow specifying arbitrary parameters in testdouble.js, it just isn’t the default:

//tell td to ignore extra arguments entirely
td.when(stub('hello'), { ignoreExtraArgs: true }).thenReturn('foo');

With ignoreExtraArgs: true the behavior is similar to Sinon.js

testdouble.js has built-in Promise support

While using promises with Sinon.js is not complicated, testdouble.js has built-in methods for returning and rejecting promises.

var stub = sinon.stub();
stub.returns(Promise.resolve('foo'));
//or
stub.returns(Promise.reject('foo'));

vs.

var stub = td.function();
td.when(stub()).thenResolve('foo');
//or
td.when(stub()).thenReject('foo');

Note: it’s possible to include similar convenience functions in Sinon 1.x using sinon-as-promised. Sinon 2.0 and newer include promise support in the form of stub.resolves and stub.rejects

testdouble.js’ callback support is more robust

Both Sinon and testdouble provide an easy way to have a stubbed function call a callback. However, they have some differences in how they work.

Sinon uses stub.yields to have the stub call the first function it receives as a parameter.

var x = sinon.stub();
x.yields('a', 'b');

//callback1 is called with 'a' and 'b'
x(callback1, callback2);

testdouble.js defaults to a node-style pattern, where the callback is assumed to be the last parameter. You also do not have to specify it when rehearsing the invocation:

var x = td.function();
td.when(x(td.matchers.anything())).thenCallback('a', 'b');

//callback2 is called with 'a' and 'b'
x(callback1, callback2);

The thing that makes testdouble’s callback support more robust is you can easily define the behavior for scenarios with multiple callbacks, or where the callbacks are in a different order.

Suppose we want to instead call callback1

var x = td.function();
td.when(x(td.callback, td.matchers.anything())).thenCallback('a', 'b');

//callback1 is called with 'a' and 'b'
x(callback1, callback2);

Notice we passed td.callback as the first parameter to the function in td.when. This tells testdouble which parameter is the callback we wish to use.

With Sinon, it’s possible to change the behavior as well:

var x = sinon.stub();
x.callsArgWith(1, 'a', 'b');

//callback1 is called with 'a' and 'b'
x(callback1, callback2);

In this case, we use callsArgWith instead of yields. We have to provide the specific index of the call for it to work, which can be a bit fiddly especially on functions with many parameters.

What if we want to call both callbacks with some values?

var x = td.function();
td.when(x(td.callback('a', 'b'), td.callback('foo', 'bar'))).thenReturn();

//callback1 is called with 'a' and 'b'
//callback2 is called with 'foo' and 'bar'
x(callback1, callback2);

With Sinon, this is not possible at all. You can chain multiple calls to callsArgWith, but it will only ever call one of them.

testdouble.js has built-in module replacement

In addition to being able to replace functions using td.replace, testdouble lets you replace entire modules.

This is mainly useful in situations where you have a module which directly exports a function which you need to replace:

module.exports = function() {
  //do something
};

If we want to replace this with testdouble, we can use td.replace('path/to/file'), for example…

var td = require('testdouble');

//assuming the above function is in ../src/myFunc.js
var myFunc = td.replace('../src/myFunc');

myFunc();

td.verify(myFunc());

While Sinon.js can replace functions which are members of some object, it cannot replace a module in a similar fashion to this. To do this when using Sinon, you need to use another module such as proxyquire or rewire

var sinon = require('sinon');
var proxyquire = require('proxyquire');
var myFunc = proxyquire('../src/myFunc', sinon.stub());

Another thing worth noticing about module replacement is testdouble.js replaces the entire module automatically. If it’s a function export like in the example here, it replaces the function. If it’s an object containing several functions, it replaces all of them. Constructor functions and ES6 classes are also supported. Both proxyquire and rewire require you to individually specify what to replace and how.

testdouble.js is missing some of Sinon’s helpers

If you’re using Sinon’s fake timers, fake XMLHttpRequest or fake server, you’ll notice they are missing from testdouble.

Fake timers are available as a plugin, but XMLHttpRequests and Ajax functionality needs to be handled in a different way.

One easy solution is to replace the Ajax function you’re using, such as $.post:

//replace $.post so when it gets called with 'some/url',
//it will call its callback with variable `someData`
td.replace($, 'post');
td.when($.post('some/url')).thenCallback(someData);

Cleaning up after tests is easier with testdouble.js

A common stumbling block for beginners with Sinon.js tends to be cleaning up spies and stubs. The fact that Sinon provides three different ways of doing it doesn’t help.

it('should test something...', function() {
  var stub = sinon.stub(console, 'log');
  stub.restore();
});

or:

describe('something', function() {
  var sandbox;
  beforeEach(function() {
    sandbox = sinon.sandbox.create();
  });

  afterEach(function() {
    sandbox.restore();
  });

  it('should test something...', function() {
    var stub = sandbox.stub(console, 'log');
  });
});

or:

it('should test something...', sinon.test(function() {
  this.stub(console, 'log');

  //with sinon.test, the stub cleans up automatically
}));

Typically the sandbox and sinon.test methods are recommended in practice, as otherwise it’s very easy to accidentally leave stubs or spies in place, which can then cause problems in other tests. This can result in hard to track cascading failures.

testdouble.js only provides one way of cleaning up your test doubles: td.reset(). The recommended way is to call it in an afterEach hook:

describe('something', function() {
  afterEach(function() {
    td.reset();
  });

  it('should test something...', function() {
    td.replace(console, 'log');

    //the replaced log function gets cleaned up in afterEach
  });
});

This greatly simplifies both set up of test doubles and cleaning up after tests, reducing the likelihood of hard to track bugs.

Pros and Cons

We’ve looked at the functionality in both libraries now. They both offer a similar feature set, but they have a somewhat different design philosophy from each other. Can we break this down into pros and cons?

Let’s first talk about Sinon.js. It provides some additional features over testdouble.js, and some aspects of it are more configurable. This affords it some increased flexibility in more special testing scenarios. Sinon.js also uses language more familiar to those coming from other languages — concepts such as spies, stubs and mocks exist in varying libraries and are discussed in testing related books as well.

The downside of this is added complexity. While its flexibility allows experts to do more things, it also means some tasks are more complicated than in testdouble.js. For those new to the concept of test doubles, it can also have a steeper learning curve. In fact, even someone like me who is very familiar with it can have trouble elaborating some of the differences between sinon.stub and sinon.mock!

testdouble.js instead opts for a somewhat simpler interface. Most of it is reasonably straightforward to use, and feels more intuitive for JavaScript, while Sinon.js can sometimes feel like it was designed with some other language in mind. Thanks to this and some of its design principles, it can be easier to pick up for beginners, and even experienced testers will find many tasks simpler to do. For example, testdouble uses the same API for both setting up test doubles and verifying the results. It can also be less error prone due to its simpler clean-up mechanism.

testdouble’s biggest problems are caused by some of its design principles. For example, the total lack of spies can make it unusable for some who prefer using them instead of stubs. This is something that is very much a matter of opinion, and you may not find a problem at all. Apart from this, testdouble.js is offering some serious competition to Sinon.js despite being a much more recent entry.

Feature by feature comparison

Below is a feature by feature comparison:

Feature Sinon.js testdouble.js
Spies Yes No
Stubs Yes Yes
Delayed stub results No Yes
Mocks Yes Yes1
Promise support Yes (in 2.0+) Yes
Time helpers Yes Yes (via plugin)
Ajax helpers Yes No (replace function instead)
Module replacement No Yes
Built-in assertions Yes Yes
Matchers Yes Yes
Custom matchers Yes Yes
Argument captors No2 Yes
Proxy test doubles No Yes
  1. testdouble.js technically doesn’t have mocks in the way Sinon.js has them. However, since mocks in Sinon are essentially objects which contain stubs and verifications, a similar effect can be achieved by using td.replace(someObject)
  2. Some similar effects to argument captors can be achieved by using stub.yield (not to be confused with stub.yields)

Summary and Conclusion

Both Sinon.js and testdouble.js provide a fairly similar set of functionality. Neither of them is clearly superior in this sense.

The biggest differences between the two are in their API. Sinon.js is perhaps slightly more verbose, while providing a lot of options on how to do things. This can be both its blessing and curse. testdouble.js has a more streamlined API, which can make it easier to learn and use, but due to its more opinionated design, some may find it problematic.

So which one is right for me?

Do you agree with testdouble’s design principles? If yes, then there’s no reason not to use it. I’ve used Sinon.js in many projects, and I can safely say testdouble.js does at least 95% of everything I’ve done with Sinon.js, and the remaining 5% is probably doable via some easy workaround.

If you’ve found Sinon.js difficult to use, or are looking for a more “JavaScripty” way to do test doubles, then testdouble.js might also be for you. Even as someone who has spent a lot of time learning to use Sinon, I’m inclined to recommend trying testdouble.js and seeing if you like it.

Certain aspects of testdouble.js can however cause headaches for those who know Sinon.js or otherwise are veteran testers. For example the total lack of spies can be a deal breaker. For experts and those who want the maximum amount of flexibility, Sinon.js is still a great choice.

If you want to learn more about how to use test doubles in practice, check out my free Sinon.js in the Real-World guide. Although it uses Sinon.js, you can apply the same techniques and best practices with testdouble.js as well.

Questions? Comments? Are you using testdouble.js already? Would you consider giving it a try after reading this article? Let me know in the comments below.

This article was peer reviewed by James Wright, Joan Yin, Christian Johansen and Justin Searls. Thanks to all of SitePoint’s peer reviewers for making SitePoint content the best it can be!

  • Александра Калинина

    calledWithExactly function could help in case:

    “`
    var stub = sinon.stub();
    stub.withArgs(‘hello’).returns(‘foo’);

    console.log(stub(‘hello’, ‘world’)); //output: ‘foo’

    sinon.assert.calledWithExactly(stub, ‘hello’); //error
    “`

    • That does indeed help with some cases, but for example `withArgs` is always “loose” with its parameters :)

Recommended
Sponsors
Get the latest in JavaScript, once a week, for free.