Unit testing, worth the effort?

[fristi]

While reading up on some things I coincidental stumbled on something called unit testing like Simpletest, phpunit or phpt.
I got interested and started reading some more. But all the examples they keep giving in tutorials seem so stupid to me. You need to write an entire test script just to test if a scripts outputs 'hello world'...
What is the point in that? If I expect 'hello world' I just run the script and see if it that actually comes out.

But since there are 3 well known test frameworks I probably am missing something.
So I was wondering what the sitepoint users think of this?

Do you Unit test? why/ why not?
Any experience with the above test frameworks? Any preferences? why?

[wysiwyg]

For me, unit testing is about feeling comfortable, basically. When I don't have unit tests, I'm a little scared of making changes in the code, because I'm not sure if it'll introduce bugs or not. But if I have a lot of unit tests, I know that they will probably break if I introduce a bug with my code changes, and that comforts me.
Unit tests can never catch all bugs though, but it's better to have unit tests that catches 80% of the bugs, than having to find all the bugs by hand.

[Amenthes]

A project is hardly ever going to have just one function returning a "Hello world"
string. It will much more likely consist of several classes and functions, inter-
connected in weird ways (most of the time). In the latter case you want to be
sure that changing the implementation of one component will not introduce bugs
in all the other components that depend on the former. I could give you an
example but it will probably be pretty long if it is to be different from what you
have seen already.

And you're right about testing it by hand, but we're programmers, and when's
too much to do by hand we automate things. So test automation is no exception.
It's much more pleasant to see green when all tests pass than having to manually
run scripts or hitting F5 in the browser for different pages, open up Firebug,
because the problem might just as well be in the HTTP layer. Who knows when
there are so many variables? That's why we should strive for testing units of
code.

I guess a good analogy would be rock climbing. Every once in a while you have
to hook up some equipment to catch you in case of falling. I made up this
analogy right now, so it may not be perfect, but that's how I see it. You climb
a little distance (write some code) and then place some safety equipment in
place (unit tests) then start climbing again. If something goes wrong, you only
fell as much as the last safety place. (Sorry if not all terms are right, I don't
know too much English terms about this field).

But, much more effective is writing the tests first. Writing tests first forces you
to think about the design in a pragmatic way. You should read Misko
Hevery's blog posts about writing testable code. If you're like me, you'll be
enlightened.

Actually, as an example of TDD, I've recently come across some slides by
Robert C. Martin (Uncle Bob) in which he demonstrates (performs a so-called
coding kata) TDD: The Bowling Game Kata

One more thing that really got me started into TDD, although I've been trying
to do it for several months, was to gather around with some people and perform
what is called a coding dojo or code retreat (haven't found any good resources
about this). At that point I realized that I already knew how to do TDD, so no
more procrastination because "I'm not yet doing it right".

But, I talked too much already. Hopefully I made myself clear, otherwise I can
write some example to give you an idea of what I mean, if you so wish.

[webaddictz]

You need to write an entire test script just to test if a scripts outputs 'hello world'... What is the point in that? If I expect 'hello world' I just run the script and see if it that actually comes out. [...] Any experience with the above test frameworks? Any preferences? why?

I do use Unit Testing, a lot. I can imagine that one would think it's overkill, as all examples usually do take the hello world example. In that case, writing all of the testing code will definitely take up more time than just alt-tabbing to your browser, hitting F5 and see if it correctly displays "hello world". But that's not the point people are trying to make. There are two rather large differences between a simple "hello world" example and the real life implementations of Unit Testing: the first is that your application will (hopefully) be more complicated than simply displaying a single string.

When you write a test case, you can execute it many times, for example, before each and every commit. It's automated, and when you've got hundreds of lines of code and you wish to check *every* functionality, I'd rather see php unit than clicking my brains out in a browser. Not to mention: you tend to forget to test, or forget at least one test, and as you know: the one you've skipped when testing manually, is always the first one that your customer will complain about.

Running tests automatically is *much* faster, less error-prone and better all-round. Just imagine yourself checking hundreds or thousands of lines of "hello world" and think who can do it faster and more accurately: the computer, or you? Do you really want to test *everything* manually again when you've implemented a change, while you could simple do "run tests"?

The first major difference ought to be enough to have convinced you, but there's another difference to be pointed out: when you're alt-tabbing, F5'ing and just checking manually if there's a string saying "hello world", you don't know how it got there. You don't know if all objects did what they actually had to do, you only know that in the end, it has the correct result. You want to know if your units (single object, or closely related set of objects) actually communicate in the correct manner, not just that they communicate. That's important. When running a test on a single unit, you know exactly what fails, there's no guessing involved. If there are 300 objects working to output "hello world" and it doesn't display "hello world", which object was at fault then? Unit Test will give you an instantaneous feedback of what was wrong. Manual testing would mean that you'd be debugging longer.

But wait: buy two, get one free! When you're accustomed to unit testing, in a fashion that you write the tests before you start writing your code, which is also known as Test Driven Design, you'll notice that that actually happens. The tests will start driving your design. Because you're using the interface before it's even written, you'll only ever write the code that you actually need, instead of writing code that you think it could be necessary to write. Besides, it'll make for a more pragmatic API. Start using it, and you'll start loving it. Promise.

[Cups]

@fristi - I can remember thinking the same thing as you. Don't worry about it, it seems to be part of the process. Leave it on the back burner in your mind for a little while longer.

The benefits of Unit testing and TDD came to me slowly too.

Next time you work on something which seemed fairly simple and then find you are spend days testing and retesting, f5-ing screen after screen just because you need to check whether all of the possible equations are covered, how your classes handle failure etc - THAT is when a light bulb may come on for you as your brain makes the connection between the actual total of mind-numbing hours lost vs the (perceived) cost of unit testing.

That was the first thing that convinced me about unit testing. I took a pen out and added up the hours I'd lost (it was an entire weekend, actually).

The second thing is when I realised the TDD process helped me out of "analysis paralysis".

I often knew roughly what I wanted a class to do, but had no idea how to start to implement it (OMG, which pattern will this be?)

TDD is a superb way of starting to write code, even just to write a stub of a class is motivation to get you going. You don't need to work to strict UML drawings or plans, you can just test and fix, test and fix and a solution evolves.

Its very liberating once you have got your setup sorted out, I found this to be a critical step - make it as easy as possible to use TDD by default.

Put your unit testing folder in your include_path directive.
Have a simple multi-test loader.

<?php
require_once('simpletest/unit_tester.php');
require_once('simpletest/mock_objects.php');
require_once('simpletest/reporter.php');

    $test = &new GroupTest('Residents files');

    $test->addTestFile('testProfiler.php');
    $test->addTestFile('testWarden.php');

    $test->run(new HtmlReporter());
?>

[imaginethis]

You know I honestly don't feel comfortable enough with unit testing to introduce into my day to day projects. I'm the only programmer in my office and my code is often repetitive with a lot of copy and pasting of pre-existing classes that have been written for specific and simple purposes. I have my own specific way of testing and developing that works for all the project I take on. For larger projects and within teams that may change, but for now I'm happy keeping things simple.

[kyberfabrikken]

Put your unit testing folder in your include_path directive.
Have a simple multi-test loader.
*snip*

There's a much simpler way:

require_once 'simpletest/autorun.php'; 

Will run all tests declared in the file. Plus it gives you command line options to run a single test selectively.

[kyberfabrikken]

You know I honestly don't feel comfortable enough with unit testing to introduce into my day to day projects. I'm the only programmer in my office and my code is often repetitive with a lot of copy and pasting of pre-existing classes that have been written for specific and simple purposes. I have my own specific way of testing and developing that works for all the project I take on. For larger projects and within teams that may change, but for now I'm happy keeping things simple.

It's a good idea not to make it an all-or-nothing approach. I certainly don't write tests for everything, but I'll usually have at least a few very high-level functional tests (also known as smoke tests). They aren't particularly precise, except that they'll warn me if I accidentally break something. If you're building a web app, Simpletest's WebTestCase can be used for this.

[imaginethis]

It's a good idea not to make it an all-or-nothing approach. I certainly don't write tests for everything, but I'll usually have at least a few very high-level functional tests (also known as smoke tests). They aren't particularly precise, except that they'll warn me if I accidentally break something. If you're building a web app, Simpletest's WebTestCase can be used for this.

Kool. I'll check them out, though I highly doubt I'll adapt my paradigm any time soon ^_^.

[Cups]

Unit testing, worth the effort?

Well as I described, TDD is probably most beneficial if you are writing classes or families of classes from the ground up. That also reaps the most benefits, and to me makes the most sense.

UnitTesting existing classes is not something I have ever done, although I have occasionally used TDD to refactor old classes - but that is probably down to my lack of experience.

Another piece of advice I'd offer is not to try and put all of the 'TDD best practice' advice to work straight off. This was especially true for me when it came to testing the layer which talks to my database.

When starting out I found it curiously reassuring to setUp() and tearDown() tables in a 'test' database, this might be because everything I build seems to turn on PDO.

It might seem like overkill, but when you are trying to train your brain to think in a new way it helps to reinforce learning if you can use TDD to do the simpler tasks first.

Now I am tending to use Mock objects more often, but nothing beats being able to query the database and see what is actually in there.

[Cups]

There's a much simpler way:

require_once 'simpletest/autorun.php'; 

Will run all tests declared in the file. Plus it gives you command line options to run a single test selectively.

Thanks, I never tried that.

hmmmm ... smoke tests.

[fristi]

Wow!!!!

I would like to thank all of you for taking the time to write such impressive posts


I think I'm starting to see the importance at this point, my way of thinking needs to be adjusted a bit to these ideas. I will definitely give Unit testing a go to see how it feels. I probably won't get the hang of it straight away, but maybe one day...

[lastcraft]

Hi...

Unit testing, worth the effort?
When starting out I found it curiously reassuring to setUp() and tearDown() tables in a 'test' database, this might be because everything I build seems to turn on PDO.

For testing SQL queries, a real DB is actually the best approach. If you use an ORM, then mock that. Setting up mock to test exact pieces of SQL syntax is fiddly, fragile and error prone. Tests are there to help you refactor, not hinder it.

On the wider question, unit testing has a bunch of advantages. One advantage alone may not be enough to swing you over straight away. Here's the list.

1) Saves time. A manual test is quicker once, the second or third time you'll wished you automated it. If you catch yourself manually testing something twice, automate it. If you think someone else might want to do the same test, automate it for them and save them a ton of time. The test frameworks make this pretty light weight. Adding a test is usually just adding a method.

2) You know when you've fixed something. If you write your tests first, then you can fix it as you write it. This leads to...

2a) You don't break anything. This leads to...

2b) No debugging. OK, you still have design screw ups, but a lot of the trivial errors are stripped out.

3) You have a clear idea of what you are trying to code. If you are pairing with someone, then you both have the same idea of what you are trying to code.

4) If you write one test at a time, then you are always writing something simple. This means you make fewer mistakes. If you get test failures, write smaller tests. If get all green, then speed up by writing larger ones. If you ride the perfect edge, you go maximum speed with no errors. This means no debugging. Um...I think we covered that.

5) Changing code is easy. Try something, see what tests fail. Then either try something else, or fix the code around it. With the tests already written, this is quick. This leads to...

5a) Refactoring. This is just not feasible without a test suite. You write quick and dirty code to get going. Then you refactor. Refactoring is changing the design without changing features. Instead of design -> code -> test-> debug, you now get test-> code -> design. Notice anything missing?

6) Refactoring means you can always improve the design and it's also easy to remove features. Chop the code, then go through the tests. If it tests the old feature, chop that too. If it's a feaure you want to keep, you had a dependency. Fix it, move on. This menas your code does not just acrete over time. It turns from perishible goods into an asset.

7) The test document the code. Want to know what something does? Look at the tests (broken down by feature and API) rather then the code (chosen for brevity and transparency). This makes your asset usable by by other developers.

8) Easier installation and checkout. Load the code, run the tests. Something broken? The test tell you where.

9) Better bug reports and patch testing. Get the submitter to send you a failing test to fix. Get a patcher to send tests with their code. That way you nkow both lumps of code work with each other. You have to do this when committing code to your version control anyway.

10) Code that's alway deployable. If the test are passing, ship it. In a team, you know that as long as the repo is passing, you were the developer who broke it if the test fail on your box. YOu can run nightly builds or continuous integration to enforce this.

11) More stuff...

Really though, you have to try it. It makes coding so easy, it feels like cheating.

yours, Marcus

[Cups]

Coincidentally Marcus, I was discussing unit tests with someone another SP-er and the subject of best practices came up, well more accurately, I should probably say testing anti-patterns actually.

I am aware of books on unit test patterns, but I was musing that I would welcome a book written by you on PHP (or webapplication) unit testing patterns with simpletest or the other one.

Any chance of that ever happening?

[lastcraft]

Hi...

Any chance of that ever happening?

I'd like to. Perhaps when both kids are school age . Actaully, things are getting less hectic. I have rewritten Phemto and have had time to work on SimpleTest lately.

Anyways, back to the thread...

yours, Marcus

[fristi]

That's so cool about sitepoint, the people who actually created well known frameworks are here to advice you

I have read so many reasons on this thread that I decided to give it a go immediately on a small class I will write. So i'll start with writing my test bench first
If I stay silent it means I'm doing ok, otherwise I'll probably come back to ask for advices.

[Amenthes]

I have read so many reasons on this thread that I decided to give it a go immediately on a small class I will write. So i'll start with writing my test bench first

Hey, a word of warning. Don't write all your tests in the first place. Write
a test, write the implementation, rinse, repeat

I once made the same mistake and is awful to discover that you've written 50
test methods and you can't even get the first one pass.

There's an ABC of TDD and that ABC sounds like this: RED, GREEN, REFACTOR.
That means, you should do it all in small iterative steps. First, write the test
method that will of course fail (red) because there's no implementation. Then
write the implementation in order to get the test pass (green). This step is
very tricky, as you may be tempted to write some clever, complicated code from
the beginning. Don't do that. Write the least amount of code that could
make that test pass. Once you know your implementation passes the test you
may refactor it in order to get better speed or structure.

If you write all of your tests from the first place it's like doing big upfront designs.
Do it in small steps and all we'll be better. Check out the slides I have linked to
in my previous post.

Hope it helps.

[fristi]

thanks for the tip

[fristi]

I'm already stuck with my testing

I created an empty testfunction:

public function testParse()
{

} 

and on execution it keeps giving me this error:
Exception: templateTest -> testParse -> Unexpected PHP error [Undefined variable: allowedTokens]


when I remove the empty function completely no errors are given...

so what is simpleTest trying to do with this empty function???

[fristi]

Found the problem

Had nothing to do with SimpleTest. I was using a require_once inside a method...
replacing it with require stopped the error. Strange behavior :/

[resting]

interesting. i'm struggling with unit testing too.
if my application is written in procedural structure. how does unit testing come into place? what i'm reading now is about using it to test classes.

in fact, i'm struggling with
OOP (design patterns, abstract and interfaces, etc),
frameworks (Zend, cakephp etc),
unit testing (PHPunit, simpletest etc) ,
good practices (MVC, file naming, directory structures etc)

i'm overwhelmed by the amount of information. i think it'll take me years to master them. are there any online courses or mentors to help us out on projects?

[Amenthes]

I've used PHPUnit in order to test functions, there's not much difference between
testing classes and testing functions. Actually when testing classes we test
methods, not classes per se. So it's pretty much the same, save for the object
instantiation. And you probably have your functions organized in some kind of
module, which may act like classes:

<?php

require_once 'module/including/foo.php';

class ModuleTest extends PHPUnit_Framework_TestCase
{
    public function testFunctionFoo()
    {
        $expect = 'foo';
        $result = foo();

        $this->assertEquals($expect, $result);
    }
}

If you're using PHP 5.3 (although I doubt it) you can substitute module for
namespace and this is what we'll probably do in PHP 5.3.

[Kayarc]

So is unit testing most valuable when working on a large-scale project?

[Amenthes]

So is unit testing most valuable when working on a large-scale project?

It is always valuable, but especially when you have to maintain a project.
Small or large, does not matter that much. But if you're called six months after
you finished a project to modify some functionality, you'd want to have tests.
You want to be sure you're not introducing bugs in the old code with the new
features.

Anyway, the above is just one side of unit tests, the side that helps you check
functionality. Unit tests are also helpful in designing your application, but only
when you're doing TDD. TDD helps you keep components decoupled, which in
turn will help teams in sharing components.

[REMIYA]

Unit testing ....

... They sound great in principle...

http://www.joelonsoftware.com/items/2009/09/23.html