Testing tool call for features

[lastcraft]

Hi...

One of my most hated things when working with a library/software is to have to modify its source code or extend something from it on the FIRST day I'm using it.

You don't have to modify the source. SimpleTest is designed to be extensible exactly because I cannot see all possible uses for the tool. It is a framework after all, from which you develop your in-house testing. An example tool extension would be Arbiter (on SF).

To add exception support you need to subclass SimpleInvokerDecorator and add an assertException() method to the test case and finally override createInvoker() to wrap it in your new decorator. About a dozen lines of code all told. Because the innards of the tool are subject to change, this is not currently documented. You will have to read the source code and work it out. In particular, PHP4 errors are trapped that way.

Right now, I don't want to make major forks for PHP5 specific code due to lack of resources and anyway, PHP5 is installed on at most about 3% of live servers.

yours, Marcus

[su1d]

Also it'll be great if the "passed test displayer" can display it according to several detail levels:
- grouping only
- grouping & test cases
- grouping, test cases, and test methods
- all details (grouping, test cases, test methods, & asserts)

Not so long ago I've got almost the same idea and tried to implement it in a sort of a XUL frontend to SimpleTest.
At the moment it's fresh and lacks some reporting, but sitll usable.
Give it a try: http://dev.e-taller.net/simpletestxul/

[ceefour]

Not so long ago I've got almost the same idea and tried to implement it in a sort of a XUL frontend to SimpleTest.
At the moment it's fresh and lacks some reporting, but sitll usable.
Give it a try: http://dev.e-taller.net/simpletestxul/

Damn! This is the coolest thing on earth! This deserves to be in SimpleTest's frontpage.

Is there a better JavaScript/DOM implementation of HtmlReporter out there...? Preferably, the one that allows me to play built-in tetris while waiting for the tests to complete. ;-) [no, just kidding!]

But I'm serious... I'll post my HtmlReporter when it's a bit better.

BTW paint* methods has bugs. They require the parent paint* methods to be called, if not the [fail/pass/etc.] numbers won't get updated. The only feasible way now is to call SimpleReporter:aint* and not (parent|HtmlReporter):aint*, because that'll also actually paint, which is not what we want (unless you want to output buffer then discard it).

It is a bug IMHO, because paint* should not be tied to how the test counting works. Why should they? Weird. I'm starting to unbelieve the SimpleTest's architecture, almost like the way I have completely lost my belief to MDB2's [poor] architecture.

[ceefour]

"To add exception support you need to subclass SimpleInvokerDecorator..."
Huh? Damn... that looks complicated. Considering how exception works, I don't even have a clue how to handle that and still allow the next code to be executed when an exception happens (go figure).

"I don't want to make major forks for PHP5 specific code..."
Who said you have to? My proposal is to do conditional include *just* for this specific case, assert[No]Exception(), only because it's the most basic functionality. All things PHP 5 is later, I completely agree, but not exceptions. It's like building a bulletproof vest without testing the vest against the bullet...

[lastcraft]

Hi...

BTW paint* methods has bugs. They require the parent paint* methods to be called, if not the [fail/pass/etc.] numbers won't get updated. The only feasible way now is to call SimpleReporter:aint* and not (parent|HtmlReporter):aint*, because that'll also actually paint, which is not what we want (unless you want to output buffer then discard it).

That's exactly the kind of problem you run into when you try to inherit from concrete classes. Use the HtmlReporter for minor changes only. Use the SimpleReporter (if I remember correctly) which is the abstract class for this. Alternately wait until I have had a chance to refactor this part of the internals.

It is a bug IMHO, because paint* should not be tied to how the test counting works. Why should they?

It isn't.

Weird. I'm starting to unbelieve the SimpleTest's architecture, almost like the way I have completely lost my belief to MDB2's [poor] architecture.

Then don't use it.

yours, Marcus

[lastcraft]

Hi....

Who said you have to? My proposal is to do conditional include *just* for this specific case, assert[No]Exception(), only because it's the most basic functionality. All things PHP 5 is later, I completely agree, but not exceptions. It's like building a bulletproof vest without testing the vest against the bullet...

You don't understand decorators and yet you are telling me which features to implement in which order ? Any forking of code, with ifdefs or anything else, is duplication and *just* a lot more work. Now as it happens exceptions are queued as a feature for version 1.1 (see the roadmap on the SimpleTest mail list archives) by which time PHP5 should be more than just a side show. However the first aim was to get a version 1 out as quickly as possible and this has only just happened.

Projects take longer when you have a child and a day job.

Anyway that feature is not actually "missing" as SimpleTest is capable of it. SimpleTest is a framework upon which you build your own test tools. The flex point I described above is deliberate. It's for adding exceptions, time limits, statistical tests and other simple extensions (or removing the ones already there). It's used to build the WebTestCase for example.

If I add every feature imaginable I get a toolkit that is harder to use. It also means that less time is spent on the useful stuff, because I am implementing stuff that can be added by the user. I'd rather add a minimal implementation and a flex point than lot's of half baked features. It's a tool for programmers after all.

The architecture is a long way from perfect I freely admit. However at least criticise the genuine shortcomings rather than the parts that do actually work .

yours, Marcus

[ceefour]

"Then don't use it."

Err... considering my current state of addiction... I'm afraid that's not a viable option for me now. Maybe I'll consider not using it yesterday. ;-)

All your points are correct and I agree completely, and it's even obvious that I have misunderstood some parts of SimpleTest... add that to the fact that I'm a very newbie (but cocky :-) user. ;-)

About SimpleTest's architecture, my previous opinion were very largely biased (and raised because of sudden surprise). I admit fully that I am blind about the real "architecture" of software, I guess what I said before is just my perception of the software (it's very different to look at a building from the sidewalk as a stranger than to feel the building as the building's architect). "Decorator" is something I've never heard of... hopefully I can learn about it from you and from SimpleTest's code...

As for the 'genuine shortcomings' of SimpleTest... I can't even think of one as of now. It's amazing that SimpleTest is one of the newest members of unit testing library in PHP, yet now it's very complete and full-featured, and still maintains ease of use. I couldn't grasp PHPUnit2 even after trying to sift through some of its code (that basically means I'm stupid), yet it only took the same idiot only a few minutes to get a working, meaningful, tests under SimpleTest... While I won't say that other libraries are worse than SimpleTest, I would confidently say that SimpleTest is much better in almost all areas than the others. I also would say your excellent thoughts in its design, ease of use, and also tutorials, really help a lot. When I noticed it has a web tester, I figured, "Nah, this is too good to be true." It turns out you're the person who realized my "too good to be true" dreams... :-)

So, there are too many good things in SimpleTest that I never mentioned only because they're so obvious (and I didn't realize that anybody cares?)
My favorite is because SimpleTest's learning curve seems flat, it's not even rising, as it seems to decrease the more I learn about it. While on learning other things we're supposed to learn a new knowledge based on previous knowledge, this doesn't seem to hold true for SimpleTest. Instead, well, you provided us with a quick one-page tutorial to write a simple test case, that won't take anybody more than an hour to get used to. After that, there's no "direction" to pick, like we can learn about web tester first before using mock objects or vice versa... This is very *very* nice.

My second love to SimpleTest is because of its HtmlReporter user interface! Really, it displays simple, but useful, and elegant, interface. And IMHO it's one of the addictive factors of SimpleTest. Had SimpleTest uses a plain text output, using unit tests won't be very entertaining. Now, it's very entertaining and my current reporter has extended quite some parts of it, using flashy colors and boxes, JavaScripts, and a helpful progress bar... it's far from su1d's wonderful XUL interface but at least it demonstrates SimpleTest's addictiveness.

My point? Well... I don't really have a point. Maybe I'm just trying to say that I have flaws too (a lot of them). Marcus definitely has outsurpassed his goals developing SimpleTest. God, however, didn't seem to have high expectations when He created me.

[lastcraft]

Hi...

So, there are too many good things in SimpleTest that I never mentioned only because they're so obvious (and I didn't realize that anybody cares?)

Ok, I'm feeling guilty . Looks like I overreacted again. I must have had a stressful week and not realised .

As for the 'genuine shortcomings' of SimpleTest... I can't even think of one as of now

1) Creating the message strings in the unit tester rather than the reporter.
2) Reporting assertion successes that are never shown rather than test method successes. Luckily CPU cycles are cheap.
3) Keeping circular references only to end up with a defeated garbage collector.
4) Not having an easy way to select just one test.
5) Only being able to set up mock call sequences on single methods.
6) Having only a single intercept for the Invoker, namely run().

Probably lot's of others that I've forgotten, but that would do for starters.

yours, Marcus

[ceefour]

If you can list 6 for SimpleTest, imagine how many loads of trucks you need to list the same things for other libs.

Anyways, these "6 stuff" reminded me of one thing (warning: off-topic, for your personal use only). One day I told my [ex]girlfriend to think about 6 things she doesn't like about me, write it in a piece of paper, then hand it to me, then I'll think a way to obey her wish. I also told her I will do the same to her.

After a few days, I got exactly what I requested. 6 items. I partly forgot what they are all now but some of them are my basic [unique] behaviors. I put that paper inside my wallet so I carry them everywhere, sometimes reading it and just ignoring it. As for her, she completely forgot that I was supposed to also give her 6 items. I never did, as that was exactly the expected plan.

It's amazing how people (or our loved ones) would expect us to be although in the other hand we really don't expect them to be anything but just the way they are...