Testing Your Tests? Who Watches the Watchmen?

Share this article

Testing Your Tests? Who Watches the Watchmen?

Regardless of whether you’re working for a big corporation, a startup, or just for yourself, unit testing is not only helpful, but often indispensable. We use unit tests to test our code, but what happens if our tests are wrong or incomplete? What can we use to test our tests? Who watches the watchmen? Who watches the watchmen graffiti

Enter Mutation Testing

No, no, it’s nothing like that. Mutation Testing ( or Mutant Analysis ) is a technique used to create and evaluate the quality of software tests. It consists of modifying the tests in very small ways. Each modified version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. Mutations are bugs in our original code and analysis checks if our tests detect those bugs. In a nutshell, if a test still works after it’s mutated, it’s not a good test.

Mutation Testing with Humbug

Humbug is a mutation testing framework for PHP. In order for Humbug to be able to generate code coverage, we will have to have XDebug installed and enabled on our machine. Then, we can install it as a global tool.
composer global require 'humbug/humbug'
After this, if we run the
command, we should be able to see some of our Humbug installation information and an error indicating that we don’t have a humbug.json file.


Before we configure and use Humbug, we need a project that we can test. We will create a small PHP calculator package where we will run our unit and mutation tests. Let’s create a /Calculator folder. Inside it, let’s create our /src and /tests folders. Inside our /src folder, we will have our application code; the /tests folder will contain our unit tests. We will also need to use PHPUnit in our package. The best way to do that is using Composer. Let’s install PHPUnit using the following command:
composer global require phpunit/phpunit
Let’s create our Calculator. Inside the /src
folder, create a Calculator.php file and add the following content:

namespace package\Calculator;

class Calculator {

    public function add($a1, $a2) {
        return $a1 + $a2;

    public function subtract($a1, $a2) {
        return $a1 - $a2;

    public function multiply($a1, $a2) {
        return $a1 * $a2;

    public function divide($a1, $a2) {

        if ($a2 === 0) {
            return false;

        return $a1 / $a2;


    //This will return $a1 percent of $a2
    public function percentage($a1, $a2) {
        return ( $a1 / $a2 ) * 100;

     * PI

    //Returns the value of pi
    public function pi() {
        return pi();


    //Returns the basic logarithm in base 10
    public function log($a) {
        return log10($a);

It is a rather straightforward program. A simple calculator, with the basic arithmetic, percentage and logarithmic operations and a function to return the value of pi. Next, inside our /tests folder, let’s create the unit tests for our calculator. If you need help with unit testing in PHP, check out this tutorial. Create a CalculatorTest.php file and add the following:

use package\Calculator\Calculator;

class CalculatorTest extends PHPUnit_Framework_TestCase {

    public function testAdd() {
        $calculator = new Calculator();
        $result = $calculator->add(2, 3);
        $this->assertEquals($result, 5);

    public function testSubtract() {
        $calculator = new Calculator();
        $result = $calculator->subtract(6, 3);
        $this->assertEquals($result, 3);

    public function testMultiply() {
        $calculator = new Calculator();
        $result = $calculator->multiply(6, 3);
        $this->assertEquals($result, 18);

    public function testDivide() {
        $calculator = new Calculator();
        $result = $calculator->divide(6, 3);
        $this->assertEquals($result, 2);

This will be our initial test stack. If we run the phpunit, command we will see that it executes successfully, and our 4 tests and 4 assertions will pass. It is important that all of our tests are passing, otherwise, Humbug will fail.

Configuring Humbug

Humbug may either be configured manually, by creating a humbug.json.dist file, or automatically, by running the command:
humbug configure
Running the command will ask us for answers to some questions:
  • What source directories do you want to include? In this one we will go with src/, the directory of our source code.
  • Any directories you want to exclude from within your source directory? May be useful in some cases, like an external vendor directory that we don’t want tested. It does not apply in our current case.
  • Single test suite timeout in seconds. Let’s go with 30 seconds on this one. It is probably too much, but we want to be sure everything has had enough time to run.
  • Where do you want to store your text log? humblog.txt comes as default and we will leave it as that.
  • Where do you want to store your json log (if you need it)? The default comes empty but we will store it in humblogjson.json.
  • Generate “humblog.json.dist”? This file will, when generated, contain all the configuration values we just supplied. We can edit it manually if we want to change something.

Using Humbug

Now that we have both our application running with tests and Humbug installed, let’s run Humbug and check the results.
The result should be close to this: Humbug feedback

Interpreting Humbug results

The number of mutations created is just the number of small changes introduced by Humbug to test our tests. A killed mutant (.) is a mutation that caused a test to fail. Don’t be confused, this is a positive result! An escaped mutation (M)
is a mutation where the test still passed. This is not a positive result, we should go back to our test and check what’s missing. An uncovered mutation (S) is a mutation that occurs in a line not covered by a unit test. Fatal errors (E) and timeouts (T) are mutations that created fatal errors and mutations that create infinite loops, respectively.

What about the metrics?

The Mutation Score Indicator indicates the percentage of generated mutations that were detected. We want to aim at 100%. Mutation Code Coverage indicates the percentage of tests covered by mutations. The Mutation Score Indicator gives you some idea of how effective the tests that do exist really are. Analyzing our humbug log, we can see that we have 9 mutants not covered, and some really bad metrics. Take a look at the humblogjson.json file. This file was generated automatically just like the humblog.txt file, and contains much more detailed information on what failed, where and why. We haven’t tested our percentage, pi and logarithm functions. Also, we need to cover the case where we divide a number by 0. Let’s add some more tests to cover the missing situations:
    public function testDivideByZero() {
        $calculator = new Calculator();
        $result = $calculator->divide(6, 0);

    public function testPercentage() {
        $calculator = new Calculator();
        $result = $calculator->percentage(2, 50);
        $this->assertEquals($result, 4);

    public function testPi() {
        $calculator = new Calculator();
        $result = $calculator->pi();
        $this->assertEquals($result, pi());

    public function testLog() {
        $calculator = new Calculator();
        $result = $calculator->log(10);
        $this->assertEquals($result, 1);
This time around, 100% means that all mutations were killed and that we have full code coverage.


The biggest downside of mutation testing, and by extension Humbug, is performance. Mutation testing is a slow process as it depends on a lot of factors like interplay between lines of code, number of tests, level of code coverage, and the performance of both code and tests. Humbug also does initial test runs, logging and code coverage, which add to the total duration. Additionally, Humbug is PHPUnit specific, which can be a problem for those who are using other testing frameworks. That said, Humbug is under active development and will continue to improve.


Humbug can be an important tool for maintaining your app’s longevity. As the complexity of your app increases, so does the complexity of your tests – and having them all at 100% all the time becomes incredibly important, particularly when dealing with enterprise ecosystems. The code we used in this tutorial can be cloned here. Have you used Humbug? Do you do mutation testing another way? Give us your thoughts on all this!

Frequently Asked Questions (FAQs) about “Who Watches the Watchmen?”

What is the origin of the phrase “Who Watches the Watchmen?”

The phrase “Who Watches the Watchmen?” originates from the Latin phrase “Quis custodiet ipsos custodes?” which was coined by the Roman poet Juvenal. This phrase is often used in discussions that question the integrity and accountability of those in power. It essentially asks, “Who will guard the guards themselves?” or “Who will watch over those who watch over us?” This phrase has been used in various forms of media, including the popular graphic novel series “Watchmen” by Alan Moore, Dave Gibbons, and John Higgins.

How is the phrase “Who Watches the Watchmen?” used in the context of software testing?

In the context of software testing, “Who Watches the Watchmen?” is a metaphorical question that addresses the reliability and accuracy of tests. It questions who or what is monitoring the tests to ensure they are functioning correctly and producing accurate results. This is a critical aspect of software development as it ensures the quality and reliability of the software being developed.

What is the significance of testing the tests in software development?

Testing the tests, also known as test validation, is a crucial part of software development. It ensures that the tests are accurately measuring the functionality and performance of the software. Without test validation, there is a risk that the tests may produce false positives or negatives, leading to inaccurate assessments of the software’s quality and reliability.

How can I ensure that my tests are reliable and accurate?

Ensuring the reliability and accuracy of your tests involves several steps. First, you should thoroughly review your tests to ensure they are correctly designed and implemented. Second, you should regularly validate your tests by comparing their results with known outcomes. Finally, you should continuously monitor and update your tests to ensure they remain accurate as your software evolves.

What are some common pitfalls in software testing?

Some common pitfalls in software testing include not testing enough, testing the wrong things, and not understanding what the tests are supposed to achieve. Other pitfalls include relying too heavily on automated testing without understanding its limitations, and not reviewing and updating tests regularly.

How does the “Watchmen” graphic novel relate to software testing?

The “Watchmen” graphic novel uses the phrase “Who Watches the Watchmen?” to question the accountability and integrity of those in power. In the context of software testing, this phrase can be used to question the reliability and accuracy of the tests themselves. Just as the “Watchmen” are supposed to guard society, tests are supposed to guard the quality and reliability of software. But just as the “Watchmen” need to be watched, so too do the tests.

What is the role of a software tester?

A software tester’s role is to ensure the quality and reliability of software by designing and implementing tests. These tests are used to identify and fix bugs, verify functionality, and assess performance. A software tester must also monitor and update these tests to ensure they remain accurate as the software evolves.

How can I improve my software testing skills?

Improving your software testing skills involves continuous learning and practice. You should stay updated with the latest testing methodologies and tools, and regularly practice designing and implementing tests. You should also seek feedback on your tests and be open to learning from your mistakes.

What are some good resources for learning more about software testing?

There are many resources available for learning more about software testing. These include online courses, books, blogs, and forums. Some recommended books include “The Art of Software Testing” by Glenford J. Myers, “Software Testing: A Craftsman’s Approach” by Paul C. Jorgensen, and “Testing Computer Software” by Cem Kaner, Jack Falk, and Hung Q. Nguyen.

What is the future of software testing?

The future of software testing is likely to be heavily influenced by advancements in technology. This includes the increased use of automation and artificial intelligence in testing, as well as the development of new testing methodologies to accommodate emerging technologies such as virtual reality and blockchain. However, the fundamental principles of software testing – ensuring the quality and reliability of software – will remain the same.

Claudio RibeiroClaudio Ribeiro
View Author

Cláudio Ribeiro is a software developer, traveler, and writer from Lisbon. He's the author of the book An IDE Called Vim. When he is not developing some cool feature at Kununu he is probably backpacking somewhere in the world or messing with some obscure framework.

BrunoShumbugmutation testingOOPHPPHPphpunittddTestingunit testunit testingUnit Tests
Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week