Error Condition Testing with PHPUnit

Let’s say you’re maintaining code that uses PHP’s native trigger_error() function to log error information. Let’s also say that you’re in the process of using PHPUnit to write unit tests for that code. If you refer to the PHPUnit manual, there’s a section that deals with testing for error condition. It describes how PHPUnit implements its own error handler that converts errors, warnings, and notices into exceptions and that catching those exceptions is how you should handle testing for these types of errors. However, depending on what your code looks like, it’s possible that you’ll run into a problem with PHPUnit’s approach to this. This article will detail what this problem is, how it impacts your ability to test your code, and how to go about solving it.

What’s the Problem?

Errors and exceptions behave in fundamentally different ways. Of particular relevance to this article is the fact that code execution can continue at the point immediately after trigger_error() if the error level constant passed to it is not indicative of a fatal error. When an exception is thrown, execution will continue at the beginning of a catch block found to correspond to the class of that exception, which may or may not be immediately after the point at which the exception is thrown. Let’s look at some examples of these behaviors. First, errors.

<?php
error_reporting(E_ALL | E_STRICT);
echo "Before warningn";
trigger_error("Danger Will Robinson!", E_USER_WARNING);
echo "After warningn";

You’ll get the following output if you run the above code:

Before warning
PHP Warning:  Danger Will Robinson! in /home/matt/error_handler.php on line 4
After warning

From this we see that the echo statement after the trigger_error() call is executed. Now, exceptions.

<?php
try {
    echo "Before exceptionn";
    throw new Exception("Danger Will Robinson!");
    echo "After exceptionn";
}
catch (Exception $e) {
    echo "In catch blockn";
}

And the output:

Before exception
In catch block

In contrast to the example case for errors, the code after the exception was thrown is not executed. Because PHPUnit converts errors to exceptions, errors behave the same way in unit tests as exceptions do. Any code that follows an error being triggered will not executed while it is being tested. Here’s another example:

<?php
function foo($param) {
    if (is_string($param)) {
        trigger_error(__FUNCTION__ . " no longer supports strings, pass an array", E_USER_NOTICE);
    }
    // do useful stuff with $param
    ...
}

With error-to-exception conversion, there’s no way to test if useful stuff is done with $param because that code will never be executed when the error is converted into an exception.

Side Effects of PHPUnit’s Behavior

This error-to-exception conversion causes differences from how the code will behave in development and testing than how it will behave in production. Here’s an example:

<?php
function error_handler($errno, $errstr) {
    throw new Exception($errstr);
}
set_error_handler("error_handler");

try {
    trigger_error("Danger Will Robinson!", E_USER_WARNING);
}
catch (Exception $e) {
    var_dump(error_get_last());
}
restore_error_handler();
trigger_error("Danger Will Robinson!", E_USER_WARNING);
var_dump(error_get_last());

Here’s its output:

NULL
PHP Warning:  Danger Will Robinson! in /home/matt/exception_converter.php on line 16
array(4) {
  ["type"]=>
  int(512)
  ["message"]=>
  string(21) "Danger Will Robinson!"
  ["file"]=>
  string(59) "/home/matt/exception_converter.php"
  ["line"]=>
  int(14)
}

The first var_dump() call, during which the custom error handler that converts errors to exceptions is in effect, outputs NULL. The second var_dump() call, during which PHP’s default error handler is in effect, outputs information about the error that was triggered. Note that it’s not because a custom error handler is used that the first var_dump() call outputs NULL, but because that error handler throws an exception. If the error handler shown in this example did not do that, the first var_dump() call would have the same output as the second.

The Solution

We need a solution that allows for the execution of code being tested to continue while still allowing us to check that an error condition was raised. As above examples showed, allowing code execution to continue can be done using a custom error handler that doesn’t convert errors to exceptions. What this error handler should do instead is capture error information for later analysis with assertions. Here’s what this might look this:

<?php
class MyTest extends PHPUnit_Framework_TestCase
{
    private $errors;

    protected function setUp() {
        $this->errors = array();
        set_error_handler(array($this, "errorHandler"));
    }

    public function errorHandler($errno, $errstr, $errfile, $errline, $errcontext) {
        $this->errors[] = compact("errno", "errstr", "errfile",
            "errline", "errcontext");
    }

    public function assertError($errstr, $errno) {
        foreach ($this->errors as $error) {
            if ($error["errstr"] === $errstr
                && $error["errno"] === $errno) {
                return;
            }
        }
        $this->fail("Error with level " . $errno .
            " and message '" . $errstr . "' not found in ", 
            var_export($this->errors, TRUE));
    }

    public function testDoStuff() {
        // execute code that triggers a warning
        $this->assertError("Message for the expected error",
            E_USER_WARNING);
    }
}

setUp(), which is run before each test method, handles setting up the error handler which is just another method in the same class that stores information about each error in an array. Other method like assertError() are then used by test methods like testDoStuff() to perform assertions against that error information and output relevant debugging information, like what errors were triggered compared to what errors were expected. Other useful types of assertions include logical inversions (i.e. asserting that a specific error was not triggered), checking for errors with messages that match regular expressions, or checking the number of errors triggered.

Conclusion

In instances where you don’t care about testing that the logic following a triggered error is still executed, PHPUnit’s default behavior is perfectly suitable for your needs. However, it’s important that you be aware of the implications of that behavior. In cases where you do care about the execution of such logic, it’s equally important that you know how to supplement PHPUnit’s functionality to facilitate accurate testing of your code is conditions as close to those of your production environment as is feasible. Image via Fotolia

Frequently Asked Questions (FAQs) about Testing Error Conditions with PHPUnit