Results 1 to 5 of 5
Mar 5, 2006, 12:06 #1
Rails testing semantics are all muddled
I've posted this on the rails mailing list but I thought I'd post it here too.
Yes, I believe it is and I think it is dangerous, and confusing to people new to the concepts of testing.
I've mentioned a point similar to this on here before, but with the announcement of Rails "integration" test, I'm worried that somebody somewhere is getting their types of tests worryingly mixed up.
Lets start with what we had in the beginning...unit tests (for models) and "functional" tests (for controllers). This was wrong from the start, because Rails' idea of functional tests were in fact just UNIT tests for controllers (or specifically, single actions in a controller). A unit test being, a test that tests a small unit functionality at the code level - it makes sure that that unit of code does what it is supposed to do. In a Rails app, a unit might be a single Model or it might be a single action on a controller.
So what is functional testing? Functional testing is another name for ACCEPTANCE testing. What is ACCEPTANCE testing? I will quote from this article:
Acceptance testing, also known as black-box and functional testing, is a way of testing and verifying that an application works according to functional, non-functional, and other important stakeholder requirements. Acceptance tests complement unit and integration tests, which are usually written using xUnit frameworks. Acceptance tests can also be written using a programming language, but Selenium and similar tools like Fitnesse also support tests written using a tool-specific document format.
Acceptance tests differ from unit and integration tests in the following ways:
* The application is tested as a whole, end-to-end entity, not just one class or a set of classes as it is with unit and integration tests.
* Acceptance tests are performed against the user interface using, for example, a browser against a Web application interface.
* The person writing the test cases does not necessarily know about the internals of the application, hence the designation of black-box testing. Non-technical users can write acceptance tests.
Acceptance tests are black box system tests. Each acceptance test represents some expected result from the system. Customers are responsible for verifying the correctness of the acceptance tests and reviewing test scores to decide which failed tests are of highest priority. Acceptance tests are also used as regression tests prior to a production release.
A user story is not considered complete until it has passed its acceptance tests. This means that new acceptance tests must be created each iteration or the development team will report zero progress.
The reason I bring this up is because I really believe the Rails developers should look at clarifying and renaming their tests to get the semantics correct. How can you expect testing newbies to start testing their apps when the framework they are using can't even get the names of their tests correct? So what do I propose:
* Rails' existing "functional" tests should be correctly referred to as unit tests, which is all they are. For organisational purposes I can see the advantage of breaking up your model and controller unit tests as the unit testing strategy for these two different types of object are usually different.
* Rails' new "integration" tests are renamed to functional or acceptance tests, which is what they really are. They should be seen as a built-in alternative to using a browser-based acceptance testing tool like Watir or Selenium. In fact, Rails' new built-in Integration tests could be compared to something like Selenium in "driven" mode instead of test-runner mode. Personally, I feel that for web apps Selenium in test-runner mode is more appropriate, but thats just my preference.
* Integration tests are left to the user to put together if they feel they need to use them. Their typical usage would be if there are several models that need to work together to produce results.
To support the above the tests folder would probably look something like this:
I'd like to hear people's thoughts on the above.