Error Condition Testing with PHPUnit

Let’s say you’re maintaining code that uses PHP’s native trigger_error() function to log error information. Let’s also say that you’re in the process of using PHPUnit to write unit tests for that code.

If you refer to the PHPUnit manual, there’s a section that deals with testing for error condition. It describes how PHPUnit implements its own error handler that converts errors, warnings, and notices into exceptions and that catching those exceptions is how you should handle testing for these types of errors.

However, depending on what your code looks like, it’s possible that you’ll run into a problem with PHPUnit’s approach to this. This article will detail what this problem is, how it impacts your ability to test your code, and how to go about solving it.

What’s the Problem?

Errors and exceptions behave in fundamentally different ways. Of particular relevance to this article is the fact that code execution can continue at the point immediately after trigger_error() if the error level constant passed to it is not indicative of a fatal error. When an exception is thrown, execution will continue at the beginning of a catch block found to correspond to the class of that exception, which may or may not be immediately after the point at which the exception is thrown.

Let’s look at some examples of these behaviors. First, errors.

<?php
error_reporting(E_ALL | E_STRICT);
echo "Before warningn";
trigger_error("Danger Will Robinson!", E_USER_WARNING);
echo "After warningn";

You’ll get the following output if you run the above code:

Before warning
PHP Warning:  Danger Will Robinson! in /home/matt/error_handler.php on line 4
After warning

From this we see that the echo statement after the trigger_error() call is executed.

Now, exceptions.

<?php
try {
    echo "Before exceptionn";
    throw new Exception("Danger Will Robinson!");
    echo "After exceptionn";
}
catch (Exception $e) {
    echo "In catch blockn";
}

And the output:

Before exception
In catch block

In contrast to the example case for errors, the code after the exception was thrown is not executed. Because PHPUnit converts errors to exceptions, errors behave the same way in unit tests as exceptions do. Any code that follows an error being triggered will not executed while it is being tested.

Here’s another example:

<?php
function foo($param) {
    if (is_string($param)) {
        trigger_error(__FUNCTION__ . " no longer supports strings, pass an array", E_USER_NOTICE);
    }
    // do useful stuff with $param
    ...
}

With error-to-exception conversion, there’s no way to test if useful stuff is done with $param because that code will never be executed when the error is converted into an exception.

Side Effects of PHPUnit’s Behavior

This error-to-exception conversion causes differences from how the code will behave in development and testing than how it will behave in production. Here’s an example:

<?php
function error_handler($errno, $errstr) {
    throw new Exception($errstr);
}
set_error_handler("error_handler");

try {
    trigger_error("Danger Will Robinson!", E_USER_WARNING);
}
catch (Exception $e) {
    var_dump(error_get_last());
}
restore_error_handler();
trigger_error("Danger Will Robinson!", E_USER_WARNING);
var_dump(error_get_last());

Here’s its output:

NULL
PHP Warning:  Danger Will Robinson! in /home/matt/exception_converter.php on line 16
array(4) {
  ["type"]=>
  int(512)
  ["message"]=>
  string(21) "Danger Will Robinson!"
  ["file"]=>
  string(59) "/home/matt/exception_converter.php"
  ["line"]=>
  int(14)
}

The first var_dump() call, during which the custom error handler that converts errors to exceptions is in effect, outputs NULL. The second var_dump() call, during which PHP’s default error handler is in effect, outputs information about the error that was triggered.

Note that it’s not because a custom error handler is used that the first var_dump() call outputs NULL, but because that error handler throws an exception. If the error handler shown in this example did not do that, the first var_dump() call would have the same output as the second.

The Solution

We need a solution that allows for the execution of code being tested to continue while still allowing us to check that an error condition was raised. As above examples showed, allowing code execution to continue can be done using a custom error handler that doesn’t convert errors to exceptions. What this error handler should do instead is capture error information for later analysis with assertions. Here’s what this might look this:

<?php
class MyTest extends PHPUnit_Framework_TestCase
{
    private $errors;

    protected function setUp() {
        $this->errors = array();
        set_error_handler(array($this, "errorHandler"));
    }

    public function errorHandler($errno, $errstr, $errfile, $errline, $errcontext) {
        $this->errors[] = compact("errno", "errstr", "errfile",
            "errline", "errcontext");
    }

    public function assertError($errstr, $errno) {
        foreach ($this->errors as $error) {
            if ($error["errstr"] === $errstr
                && $error["errno"] === $errno) {
                return;
            }
        }
        $this->fail("Error with level " . $errno .
            " and message '" . $errstr . "' not found in ", 
            var_export($this->errors, TRUE));
    }

    public function testDoStuff() {
        // execute code that triggers a warning
        $this->assertError("Message for the expected error",
            E_USER_WARNING);
    }
}

setUp(), which is run before each test method, handles setting up the error handler which is just another method in the same class that stores information about each error in an array. Other method like assertError() are then used by test methods like testDoStuff() to perform assertions against that error information and output relevant debugging information, like what errors were triggered compared to what errors were expected.

Other useful types of assertions include logical inversions (i.e. asserting that a specific error was not triggered), checking for errors with messages that match regular expressions, or checking the number of errors triggered.

Conclusion

In instances where you don’t care about testing that the logic following a triggered error is still executed, PHPUnit’s default behavior is perfectly suitable for your needs. However, it’s important that you be aware of the implications of that behavior.

In cases where you do care about the execution of such logic, it’s equally important that you know how to supplement PHPUnit’s functionality to facilitate accurate testing of your code is conditions as close to those of your production environment as is feasible.

Image via Fotolia

Free book: Jump Start HTML5 Basics

Grab a free copy of one our latest ebooks! Packed with hints and tips on HTML5's most powerful new features.

  • boen_robot

    Wait… doesn’t PHPUnit have a configuration option that allows you to not converting errors to exceptions? What am I missing?

    • http://damiengrass.com Damien

      I believe so, I read not a few days ago how it can be done with:

      PHPUnit_Framework_Error_Warning::$enabled = FALSE;

      But I don’t think it works with old versions.

    • http://matthewturland.com Matthew Turland

      @Damien Yes, it looks like that’s been available since around PHPUnit 3.3.0. https://github.com/sebastianbergmann/phpunit/commit/607abedfa26cc2c9c6c2ad9f7c6d8b07dbcbb1e5

      @boen_robot You can do as Damien suggests and disable error-to-exception conversion for individual error classes or use restore_error_handler() to restore the native PHP error handler so that PHPUnit’s error handler is not used. To do this, you would need to disable display_errors when running PHPUnit so that any triggered errors did not disrupt PHPUnit’s output, which may make debugging difficult. That’s another advantage of the approach I’ve described in this article: it captures error information without displaying it and makes it available for later analysis with assertions.

  • edo

    I’d much rather turn the conversion off or just use a simple:

    $result = @$object->callThatTriggersError();

    No conversion will happen there and I can still assert on te result.

    Putting a test that triggers the exceptio. before that one and using @expectedException for that alle covers all “parser error” cases. When testing legacy code that uses trigger error I’d prefer those ways o registering complex(er) custom handlers i guess :)

    • http://matthewturland.com Matthew Turland

      @edo That example may not prevent use of error_get_last() to test that an error was triggered, but it does suppress error logging (at least without the scream PECL extension) and thus prevents use of logs to debug production issues. Scott Mattocks has a great post on why this is important here: http://crisscott.com/2012/09/21/l-is-for-logging/

  • http://jtreminio.com Juan Treminio

    Hi Matt,
    I disagree with your process for testing for errors outlined here.
    You should either throw exceptions and set PHPUnit to expect an exception, *or* you can tell PHPUnit to expect a call to your error handler – which shouldn’t be simply trigger_error and var_dump! If you utilize something like monolog, you can mock out a monolog object and set the error handler to be called once when there’s a known error.

    • http://matthewturland.com Matthew Turland

      My examples in this article are only meant to showcase particular behaviors in PHP or PHPUnit, not as code that you’d actually see in quality codebases or their tests. Ideally, yes, projects would allow for error handlers to be mocked. However, many developers have to deal with codebases where this is not the case.

  • http://thenazg.blogspot.com Chuck Burgess

    Would the Example 4.12 suggestion from the 3.6 manual [1] mitigate this scenario, where using the @ silencer on your testcase’s call to the tested method would then suppress that method from being affected by an error handler?

    public function testDoStuff() {
    // execute code that triggers a warning,
    // but silence them *just* for this test
    $result = @myMethodToBeTested($args);
    $this->assertFalse($result);
    }

    [1] — http://www.phpunit.de/manual/3.6/en/writing-tests-for-phpunit.html#writing-tests-for-phpunit.exceptions.examples.TriggerErrorReturnValue.php

    • http://matthewturland.com Matthew Turland

      It would, though this is one instance where I disagree with practices recommended by the manual. The example you’re citing could fail for multiple reasons and use of the @ operator prevents me from seeing information about that reason without modifying the test code. Tests should be as informative about the reason of the failure as possible, else it significantly decreases their value for debugging.

  • http://blog.mindplay.dk Rasmus Schultz

    Two important things to note:
    1. Since PHP 5.1.0, throwing Exceptions for errors is built into PHP – you just have to enable it:

    http://us3.php.net/ErrorException

    2. You should never, ever, ever catch(Exception $e) – always and only catch Exceptions of a specific type, for example catch(ErrorException $e) … the only deviation from that rule, is if you’re going to re-throw the same exception.

    • http://matthewturland.com Matthew Turland

      Agreed on point #2. Per my earlier comment to Juan, my code wasn’t meant to be taken literally, only as an example of behaviors in PHP and PHPUnit. Thanks for your feedback.