HTML & CSS

Visual Regression Testing with PhantomCSS

Pavels Jelisejevs
Share

If you’ve done any serious development in your career you’ve likely reached the point when you realized the importance of automated testing during development. Depending on your experience, this realization might hit you in one big burst or it may gently come to you over time, but it will eventually become second nature. Automatic testing comes in many forms, from unit testing, when you test isolated pieces of code, to integration and functional testing, when you test how different parts of your system behave together. This article is not about an overview on automatic testing in general. It is about a particular and a relatively new niche referred to as visual regression testing.

Visual regression testing takes an alternative approach to testing web pages. Instead of just making sure that some element or a text value is present in the DOM, the test actually opens the page and checks if this specific block looks exactly like you want it to. Just to make sure that you picked up the difference, let me give you an example. Imagine, that you want your website to greet your visitors with a friendly message:

<div>Hello, %username%!</div>

To make sure it works, you can (and should) unit test the piece of code that produces the message, checking that it inserts the correct name. You can also write a functional test using Selenium or Protractor to see if the element is actually present on the page with the correct text. But this is not enough. We want to test not just that the text is generated correctly or appears in the DOM but to make sure that the whole element looks correct, i.e., making sure that the element is not hidden by display: none or that someone hasn’t accidentally overridden the color of the text. There are a number of tools to do that, but today we will be looking at one option in particular — PhantomCSS.

What is PhantomCSS?

PhantomCSS is a Node.js tool to perform visual regression testing. It is open source and developed by the guys at Huddle. PhantomCSS allows you to run a headless browser, open a page and take a screenshot of the whole page or a particular element on the page. This screenshot will be stored as a baseline image for future reference. Whenever you change anything on the website, you can run PhantomCSS again. It will take another screenshot and compare it to the original image. If there are no differences found, the test will pass. If, however, the screenshots don’t match, the test will fail and a new image showing the difference will be created for you to review. This approach makes this tool perfect for testing changes in CSS.

PhantomCSS is built on top of several key components:

  • CasperJS – a tool for interacting with a PhantomCSS or SlimerJS browser. It allows you to open a page and perform user interactions, such as clicking on buttons or inputting values. Additionally, CasperJS provides its own testing framework and the ability to capture screenshots of a page.
  • PhantomJS 2 or SlimerJS – two different headless browsers, either of which can be used with PhantomCSS. A headless browser is just like a normal browser without a user interface.
  • Resemble.js – a library for comparing images.

PhantomCSS can be used together with both PhantomJS and SlimerJS, but in this article, we’ll be using PhantomJS.

Let’s Take It for a Spin

Let’s set up a tiny test project to see how we can use this tool in practice. For that, we’ll need a web page to test and a simple Node.js web server for CasperJS to be able to open the page.

Setting up a Test Project

Create an index.html file with some sample content:

<!doctype html>
<html>
  <head>
    <style>
      .tag {
        color: #fff;
        font-size: 30px;
        border-radius: 10px;
        padding: 10px;
        margin: 10px;
        width: 500px;
      }

      .tag-first {
        background: lightcoral;
      }

      .tag-second {
        background: lightskyblue;
      }
    </style>
  </head>

  <body>
    <div class="tag tag-first">The moving finger writes, and having written moves on.</div>
    <div class="tag tag-second">Nor all thy piety nor all thy wit, can cancel half a line of it.</div>
  </body>
</html>

To install the web server, initialize an npm project and install the http-server package.

npm init
npm install http-server --save-dev

To run the server, let’s define a simple npm script. Just add the following scripts section to package.json

"scripts": {
  "start": "http-server"
},

Now you can run npm start from the project folder and the index page will be accessible on the default address http://127.0.0.1:8080. Start the server and leave it running for now. We’ll need it in a while.

Installing PhantomCSS

Installing PhantomCSS is easy, all you need to do is add a few dependencies to your project:

npm install phantomcss casperjs phantomjs-prebuilt --save-dev

Creating a Test Suite

Now we have everything we need to set up the first test suite. PhantomCSS test suites are created in the form of Node.js scripts where you open the required page of your website, take screenshots and compare them to the images from the previous run. We start with a simple test case based on the demo from PhantomCSS itself.

var phantomcss = require('phantomcss');

// start a casper test
casper.test.begin('Tags', function(test) {

  phantomcss.init({
    rebase: casper.cli.get('rebase')
  });

  // open page
  casper.start('http://127.0.0.1:8080/');

  // set your preferred view port size
  casper.viewport(1024, 768);

  casper.then(function() {
      // take the screenshot of the whole body element and save it under "body.png". The first parameter is actually a CSS selector
      phantomcss.screenshot('body', 'body');
  });

  casper.then(function now_check_the_screenshots() {
    // compare screenshots
    phantomcss.compareAll();
  });

  // run tests
  casper.run(function() {
    console.log('\nTHE END.');
    casper.test.done();
  });
});

The test will open http://127.0.0.1:8080/, take a screenshot of the body element and save it under screenshots/body.png.

Once we have the test itself in place, all that’s left is to define a script to run the test. Let’s add the following script to package.json next to start:

"test": "casperjs test test.js"

You can now run it by executing the following command:

npm test

The output you will see should look something like this:

Test file: test.js                                                              
# Tags
PASS Tags (NaN test)

New screenshot at ./screenshots/body_0.png

Must be your first time?
Some screenshots have been generated in the directory ./screenshots
This is your 'baseline', check the images manually. If they're wrong, delete the images.
The next time you run these tests, new screenshots will be taken.  These screenshots will be compared to the original.
If they are different, PhantomCSS will report a failure.

THE END.
WARN Looks like you didn't run any tests.                                       
npm ERR! Test failed.  See above for more details.

Since you’ve run the test for the first time, it will just create a new baseline screenshot and won’t perform any comparison. Go ahead and peek inside the screenshots folder. You should see an image like this one:

Screenshot of the whole body

This is the golden standard of how your website is supposed to look and future executions of the test will compare their results to this image.

Introducing a Regression

If you run the same test command again it will report that all tests have passed successfully:

Test file: test.js                                                              
# Tags
PASS Tags (NaN test)


PASS No changes found for screenshot ./screenshots/body_0.png

PhantomCSS found 1 tests, None of them failed. Which is good right?

If you want to make them fail, change some CSS.

THE END.
PASS 1 test executed in 0.827s, 1 passed, 0 failed, 0 dubious, 0 skipped.

This is to be expected since we haven’t changed anything on the website. Let’s break something and re-run the tests again. Try changing some styles in index.html, for example, decrease the size of the blocks to 400px. Now let’s run the test again and see what happens:

Test file: test.js                                                              
# Tags
PASS Tags (NaN test)
Failure! Saved to ./failures/body_0.fail.png


FAIL Visual change found for screenshot ./screenshots/body_0.png (11.41% mismatch)
#    type: fail
#    file: test.js
#    subject: false

PhantomCSS found 1 tests, 1 of them failed.

PhantomCSS has created some images that try to show the difference (in the directory ./failures). Fuchsia colored pixels indicate a difference between the new and old screenshots.

THE END.
FAIL 1 test executed in 1.082s, 0 passed, 1 failed, 0 dubious, 0 skipped.       

Details for the 1 failed test:

In test.js
  Tags
    fail: Visual change found for screenshot ./screenshots/body_0.png (11.41% mismatch)
npm ERR! Test failed.  See above for more details.

Several important things have happened here. First, PhantomCSS reported that the tests failed because of a mismatch for screenshot body_0.png. The mismatch is measured at 11.41%. Second, the difference between the current and the previous version was saved in the failures folder. If you open it, you will see a screenshot like this one:

Screenshot of an error

The screenshot conveniently highlights the areas that have been changed so it’s easy to spot the difference.

Accepting the Changes

Now that the difference has been highlighted, what should we do to accept the change? We should somehow be able to tell the tool that we want to stick with the reduced width of the blocks and accept the current view as the new standard. To do that, you can run the test command with an additional -- --rebase argument:

npm test -- --rebase

Note the two double dashes. It’s npm’s way of passing a parameter to the underlying command. So the following command will result in casperjs test test.js --rebase. Now that we’ve accepted the change, the previous baseline image will be replaced with the new one.

Taking it Further

Now that you’ve got the hang of the basics, you can start thinking about integrating this tool into your own workflow. I won’t get into the details of that since it is pretty project specific, but here are some questions to ponder about:

  • Are you going to run the tests against the real website, or some kind of style guide, where only separate UI elements are present?
  • Does your site have dynamic content? If yes, then changes in the content will cause the tests to break. To avoid that, you’ll need to set up a separate version of the website with static context to run the tests against it.
  • Are you going to add the screenshots into your version control? Yes, you should.
  • Are you going to take screenshots of whole pages, or separate elements?

Using this tool you can now cover the visual aspects of your website with automated tests. With your unit and functional tests already in place, this new strategy will fill a narrow gap in your testing frontier. Even if you are still new to testing — this is a good place to start!