Test Driven Development - long time users

[jplush76]

ok so I'm sold,
I just finished a small web services project that will be used to manage millions of dollars worth of orders for several companies. Every line of code is backed up with a unit test. (not literally, but each line of code is accounted for when accessing the public apis).

My stress level was low, my design came out 10X better and more flexible than I had previously envisioned. I have confidence that all the code will act according to spec. If any changes are needed I won't have to stress out because I have all the unit tests backing up my changes. I actually looked at the code and said.. thats pretty(I swear)

As this was my first step into to fully living the TDD lifestyle (red, green, refactor) it obviously went slower than coding without tests but well worth it. (thanks Marcus, Jason, and McGruff)

I was hoping some of the more established TDD users could chime in on how this style has affected your bug rate. Do you find you dramatically get fewer bugs? How about regression bugs from refactoring? Do you find even in the most stressful of release times that TDD helps manage the complexity?

[Captain Proton]

I find that I can find bugs easier and sooner in the development process with unit testing. I can find them easier because I know which test causes a unit to fail and therefore what method of using the unit results in the bug.

Before unit tests I would usually test a component manually and then just wait to find bugs, but now I can immediately test the component and re-test it automatically after I have made changes.

Also, I think TDD helps to manage complexity because you don't have to think about if a change to one part of the system will cause bugs in other parts: you can be (reasonably well) assured that it doesn't cause any bugs if your tests pass.

Most importantly though, I really love the confidence I get in my code! Perhaps it even helps me sleep better at night because I know that when a component is 'finished' and tested, it adheres to my requirements and it actually works too

[jplush76]

I've noticed that too, I'm addicted to the green bar now lol
I honestly felt more relaxed and confident after I finished the project knowing I've thrown a good amount of tests at it and it works properly in all the cases I could think of, good and bad.

[sweatje]

You have become infected. I hope it spreads

[been]

The psychological impact of the green bar is not to be underestimated, that's for sure

However, nowadays, I seem to be getting more and more comfortable with the red bar: it tells me that I'm going at the correct pace, one step at a time.
If I have a test case that runs green, I add a test and it still runs green, I get suspicious: "new tests are suppose to fail" is something that for one reason or the other has settled itself in the back of my brain.
sidenote: Refactoring against the red bar

The number of bugs has reduced, but they still go out though: I'm a solo coder most of the time and you just cannot account for everything. But the good thing is; the moment bugs are discovered, you can throw them into a test and code 'till green. Granted, it does not always go that swift, but compared to doing stuff without tests in place is just, well, no comparison

it obviously went slower than coding without tests but well worth it.

My personal experience with this is that in the beginning there's a drop in productivity, but after a project or 2 it went up. I deliver more usefull code in less time.

Do you find even in the most stressful of release times that TDD helps manage the complexity?

I think that's one of the areas were it shines. I'm taking evening courses in school, and for Java we had an assignment (a simulation of a mobile phone keyboard, incl. T9-like behaviour); we were obligated to write a few specific tests, but I simply ignored that and wrote unit tests for all the classes (and some mocks).
I started too late on the assignment and was writing my last lines of code on a sunday afternoon (we had to hand it in on monday). I never even ran the program (just the test suite), until monday morning: it barely worked, but all in all it were some minor glitches: a few extra tests and 6 hours later it fully worked as described in the specs.
I'm quite sure that if I'd stepped into the "Damn, short on time, let's skip the tests for once"-trap, I would've failed miserably.

All in all I spent about 60 hours coding and 6 hours debugging; 10/1, I like that ratio

EDIT: I discovered unit testing about 1-1,5 year ago and I never looked back. I don't think that that makes me a long time user.

[McGruff]

I've been meaning to ask how you're getting along. It's a life-changing event isn't it?

I think the more you test the more efficient you get at writing tests. It's bound to take a little longer because the testing discipline leads you to think more deeply about the behaviour of the code. It's all repaid though. Without tests, some changes would be just too scary to contemplate but, with tests, they're a breeze.

Overall there's definitely a time gain. Bugs get caught early and are relatively easy to fix with fine-grained tests to tell you exactly what is going wrong. Automated testing saves an enormous amount of time compared to trying to do it manually - you simply couldn't do this with anything like the same rigour quite apart from the time saving.

The stress factor is important. Testing gives you back control over the code and that makes programming much more fun.

I've been planning to try a FIT-style interaction with clients on my next project - another possibility opened up with testing. It'll just be something simple: requirements will be posted on a web page to which I'll add links to run the relevant tests.

[lastcraft]

Hi...

I've been planning to try a FIT-style interaction with clients on my next project - another possibility opened up with testing. It'll just be something simple: requirements will be posted on a web page to which I'll add links to run the relevant tests.

Or prhaps you could help me rerive the Arbiter project on Sourceforge ?

yours, Marcus

[arborint]

Or prhaps you could help me rerive the Arbiter project on Sourceforge

I don't want to say I told you so (I think I just did) but if you put a web interface in Arbiter rather than passing Word docs around I would be interested in reviving it as well.

[McGruff]

Hi...
Or prhaps you could help me rerive the Arbiter project on Sourceforge ?
yours, Marcus

Yes I was thinking about that. If I get the job (it's been a bit on-off so far) I'll be working with a board of trustees with little programming knowledge. I need to get an idea how easy it is to explain the concept to non-progammers and how useful they find it as a means of communication. If all goes well I'll definitely be thinking about a proper tool.

[lastcraft]

Hi Chris...

I don't want to say I told you so (I think I just did) but if you put a web interface in Arbiter rather than passing Word docs around I would be interested in reviving it as well.

The overwhelming demand from project managers (no coders) was for Word by 6 to 0. The reason for the stall is lack of leadership I think, i.e my fault, rather than a project flaw. I am currently adding file upload capability to SimpleTest to speed up the project, hence my absence.

Regarding adding web pages, I wouldn't have a problem with adding this. The trouble is that Word, more properly RTF, support is really the minumum to reach the manager audience. Developers already understand the concept of acceptance testing, the challenge is making it as seemless as possible for the clients. Clients just don't type web pages. If RTF is the minimum, and my plan is to get something running before expanding it, then the current plan is the most direct.

yours, Marcus

[jplush76]

Marcus, what have your experiences been as far as reducing bug counts and overall code stability when using unit testing throughout a project? When you get a bug report, do you write a test case for that bug to exploit it and then fix the bug while leaving the new test case?

[arborint]

The overwhelming demand from project managers (no coders) was for Word by 6 to 0.

I can believe that, but I just don't think they see the bigger picture potential of web centric project management. Managers are the worst of all employees at not sharing their information. They have their documents, their spreadsheets, their project management files ... all locked away on their computer. I have employed many managers and have always had a problem getting them to share more information.

The reason for the stall is lack of leadership I think, i.e my fault, rather than a project flaw. I am currently adding file upload capability to SimpleTest to speed up the project, hence my absence.

SimpleTest upload sounds good.

Regarding adding web pages, I wouldn't have a problem with adding this. The trouble is that Word, more properly RTF, support is really the minumum to reach the manager audience. Developers already understand the concept of acceptance testing, the challenge is making it as seemless as possible for the clients. Clients just don't type web pages. If RTF is the minimum, and my plan is to get something running before expanding it, then the current plan is the most direct.

I get the impression from looking at the Arbiter code that you are using RTF as a data source. I was originally thinking that there would be a database that would hold the information and RTF would be parsed into and generated from the database. A web interface would not require all the parsing and generating because it would be a more controlled and a closer representation of the data. That's why I thought you could build the web interface first and then add the RTF parse/generate.

[lastcraft]

Hi...

Marcus, what have your experiences been as far as reducing bug counts and overall code stability when using unit testing throughout a project?

Bugs are pretty much a rarity, but it's not just unit testing on it's own. Unit testing also makes us more willing to refactor and to take the time to craft better code. It also helps pair programming.

When you get a bug report, do you write a test case for that bug to exploit it and then fix the bug while leaving the new test case?

Nearly always and when we don't we regret it. For example, I am currently working on an adult content filter. Whenever I hear a report of a false positive or a false negative, it gets added straight into the acceptance tests.

yours, Marcus

[lastcraft]

Hi...

I can believe that, but I just don't think they see the bigger picture potential of web centric project management.

I went on a bit of a fishing expedition before the project to try and work out what would be useful. The project is an expriment to see of the idea will float. I wouldn't claim it is well researched.

Managers are the worst of all employees at not sharing their information. They have their documents, their spreadsheets, their project management files ... all locked away on their computer. I have employed many managers and have always had a problem getting them to share more information.

The value of these documents is that the tests are embedded, and the tests will be used to sign off the project. Unless these documents are exchanged, the developers cannot get requirements and the managers don't get completed acceptance tests. Both sides should have incentive, because the tool is essentially sharpening communication ad saving both sides time.

Also the manager can automatically visualise data on tests completed/added/etc by time, but taht would come later.

That's why I thought you could build the web interface first and then add the RTF parse/generate.

The coding of the project is actually pretty easy. In fact I think it has had barely 30 hours work done on it all told so far. The tricky part is the social experiment that is the act of trying this on a real project. I am trying to get this barrier as low as possible. i actually expect it to fail first time around, but in so doing some kind of greater understanding will emerge and with it a more powerful project.

yours, Marcus

[arborint]

Nearly always and when we don't we regret it. For example, I am currently working on an adult content filter. Whenever I hear a report of a false positive or a false negative, it gets added straight into the acceptance tests.

I agree. It has the added benefit of explicitly documenting the response to the bug. I would add the caveat that in some cases a bug report reveals problems that require changes to how unit testing is being done and previous assumptions.

[McGruff]

I would add the caveat that in some cases a bug report reveals problems that require changes to how unit testing is being done and previous assumptions.

Maybe I picked you up wrong but really the code should follow the test rather than vice versa. Assertions create constraints on the solution which can be tightened up until the pips squeak.

Sometimes I feel like code is something which just happens incidentally while I'm writing tests.

[DougBTX]

Maybe I picked you up wrong but really the code should follow the test rather than vice versa.

Sometimes the tests are wrong. The logic/plans/assumptions/expectations you had when you wrote the tests could be wrong, resulting later in a bug in the application itself. To fix the application, you would need to fix the test too.

That's when you need integration testing: if one unit needs to have its tests changed because something is fundamentally wrong, then you still have to have tests which make sure that code which uses the unit are updated too, if necessary.

Douglas