Request for comments about Testing sockets code

[sism82]

Hi every one. We have a couple of classes that we wrote beofore deciding to use TDD (Test Driven Development) using simpletest framework. So, this classes need to pass several tests before we continue. The more important classes are:
- AmiProxy
- ApSignalHandler

AmiProxy is intented to be daemonized later, for now is just a server that connects to Asterisk (VoIP server) and login in it. Then start listening for client connections, so, it works like a proxy between its clients ant Asterisk Server. I have to test several functionality :

- connect successfull to Asterisk
- Accept clients connections
- Relay clients connections..
- and may be more.

So, anyway, the point is, i have 2 main problems.

The simpler, does it exists in simpletest framework a Console reporter? i would like to use it instead the HtmlReporter()? this is just a dump question, because i will look into the code later, and if its not implemented i will came up with something.

The important question, or request for comments, is if is "correct" the approach im taking to test this thing. The code follows below, with a couple of comments:

require_once '../AmiProxy.php';

require_once '../ApSignalHandler.php';

/**

 * this class should test that the AmiProxy accomplish with its responsabilities

 * AmiProxy Responsabilities:

 * - Connect to Asterisk Manager through the socket localhost:5038

 * - Wait for incoming connections of clients that wish to execute one of the

 *   following commands:

 *   ::SendAction:

 *               When a client issue this command, is requesting to send

 *               the specified action to Asterisk. The action can, or cannot

 *               have an ActionID associated to it.

 *   ::LaunchEvent:

 *               This command asks the AmiProxy to send the specified event

 *               to all the clients suscribed (see below) to the event. In

 *               this way, any client can make aware of any event to all the

 *               the clients that are interested in the event.

 *   ::SuscribeEvent:

 *               In this way, a client tells to AmiProxy that we wants to be

 *               aware of the specified event. So, when the event shows up,

 *               (either because of an Asterisk Event or other client Event)

 *               the event data will be sent to the client

 *   ::UnsuscribeEvent:

 *               When a client request UnsuscribeEvent, then it wont be aware

 *               anymore of the specified event. No more event data for that

 *               event will be sent to that client.

 *    *Note about Events:

 *               events are launched either by Asterisk or other clients. Until

 *               now, we have no mechanism to avoid conflicts/collisions between

 *               Asterisk events and clients events, so its up to the developer of

 *               the client dont use events that are already taken by Asterisk

 * - Send commands to Asterisk

 * - Receive commands responses from Asterisk

 * - Receive events from Asterisk

 * - Relay events to the clients

  */

final class AmiProxyTest extends UnitTestCase

{

    private $ami_socket_uri = '';

    private $ami_username   = '';

    private $ami_password   = '';

    private $assert_file    = '/tmp/assert_AmiProxyTest';

    

    public function SetAmiProxyData($UserName, $Password, $SocketUri)

    {

        $this->ami_username   = $UserName;

        $this->ami_password   = $Password;

        $this->ami_socket_uri = $SocketUri;

    }

    

    private function ReceivePong()

    {

        file_put_contents($this->assert_file, 'true');

    }

    

    public function setUp()

    {

        @unlink($this->assert_file);

    }

    

    public function testConnectingToManager()

    {

        

        $server_pid = pcntl_fork(); /* make a fork for the server application */

        if ( $server_pid > 0 )

        {

            $client_pid = pcntl_fork(); /* make a fork for the client application */

            if ( $client_pid > 0 )

            {

                $time_left = 3;

                while ( $time_left > 0 )

                {

                    if ( file_exists($this->assert_file) )

                    {

                        break;

                    }

                    else

                    {

                        $time_left--;

                    }

                    sleep(1);

                }

                 /* KILL the child process (client and server) */

                posix_kill($server_pid, 9);

                posix_kill($client_pid, 9);

                $this->assertTrue(file_exists($this->assert_file));

            }

            else

            {

                $client = ApSignalHandler();

                $client->SetDefaultCallbackObject($this);

                $client->RegisterAction('ping');

                $client->OnActionResponse('ReceivePong'); /* Receive pong is the function to execute in case that a ping response is received*/

                $client->HandleProxyData(); /* this function enters in an infinite while to receive proxy data */

            }

        }

        else

        {

            $server = new AmiProxy($this->ami_username, $this->ami_password, $this->ami_socket_uri); /* this function enters in an infinite while, waiting for clients connections */



        }

    }

} 

Ok, here is how i think it will work.
- The test start, when enter the method testConnectingToManager, removes the flag file (/tmp/etcetc..)
- then forks the proces so the parent continue, and the first child start the AmiProxy instance and waits for clients connections
- the parent then forks again, so the parent continue, and the child (by now the second child) starts a client and try to send a command (ping) to the proxy. If the proxy is connected, then it should send the command to asterisk, asterisk wil return a response, the response sent to the client, the client will then execute the method specified by OnActionResponse(), then the method writes to the flag file.
- the parent will be cheking if the file exists for a while, if the file does not show up, we can tell that something is wrong and kill both childs and assert to false.

Any comment will be greatly appreciated.

[Ryan Wray]

Originally Posted by sism82

The simpler, does it exists in simpletest framework a Console reporter?

Of course, read: http://www.lastcraft.com/reporter_documentation.php#cli

I am feeling a bit lazy now to look at your test case, but no doubt someone will give it a gander.

[HarryF]

You're brave to being doing this in PHP but in a general sense, as far as the test goes, looks you're doing things "right", whatever that means.

Two observations that spring to mind (which may be because I haven't fully understood) - wouldn't it be better to have the server running seperately from the test (perhaps fired up by the test as a shell command - seems like trying to manage both the client and server makes the test alot more complicated? Also, if you'll be using files to exchange test results, it might be worth having some utility functions around to make locking simpler, so you don't get race conditions and the tests are easier to code.

Although it's Perl, this blog may have some useful ideas.

Otherwise, do the tests have to be written in PHP, even if your server is PHP, given you've got a "api" in the form of a network protocol and execution via the shell. Proc::Simple, for example, might make writing tests much easier and perhaps you'd even find complete test tools ready to use for this kind of problem. I just see nightmares ahead but perhaps that's me.

[sism82]

Hi!, thanks both of you for your comments and help.

Originally Posted by Ryan Wray

Of course, read: http://www.lastcraft.com/reporter_documentation.php#cli

yeah, i saw it a minuts later when i checked out the code

Originally Posted by HarryF

You're brave to being doing this in PHP

hehehe do you mean that it might be not a good idea? why?

Originally Posted by HarryF

wouldn't it be better to have the server running seperately from the test

Yep, you gotta point. I have tought that, but i have not made a decission yet. Im thinking that some kind of MockAsteriskServer will make my life easier. A dumb server that just answer simple questions may be? what the heck, really im just guessing, i havent made TDD before so im a newbie, and is kind of difficult to know what is right and what is not. But i dont like the idea of creating and killing process on the fly, but second tought that could be because im not used to programm with several process and it may be just that.

I have done some research on the web and yep, it seems that more people has problems with test on sockets. But no concrete answers. I saw in guy blog, programming in C++, a couple of thest that take that approach (using a mock server), so that option is getting interesting.

Originally Posted by HarryF

Although it's Perl, this blog may have some useful ideas.

Sure!

Originally Posted by HarryF

Otherwise, do the tests have to be written in PHP, even if your server is PHP, given you've got a "api" in the form of a network protocol and execution via the shell. Proc::Simple

Hum, it seems like a headeach to me, im not sure how much time will it take to me to learn basic perl, but dont like the idea of writing tests and code in different language, but sure i will consider this further

Originally Posted by HarryF

I just see nightmares ahead but perhaps that's me.

haha me too, but at the end the light will came up, hopefully haha

any other ideas?
im considering seriousely to launch a mock server, i guess it may work better that forking() around.

regards.

[sweatje]

Originally Posted by sism82

hehehe do you mean that it might be not a good idea? why?

PHP has notoriously bad memory management. Because the typical usage it to create a PHP process for each request, often freeing memory is delayed until the termination of the process. If you intend to have a long running daemon, this might have a significant impact on you.

[HarryF]

PHP has notoriously bad memory management. Because the typical usage it to create a PHP process for each request, often freeing memory is delayed until the termination of the process. If you intend to have a long running daemon, this might have a significant impact on you.

Good point - was actually thinking of other issues like the pcntl extension being relatively untested compared to more well used parts of PHP and that things like OS signals are better integrated in languages like Perl.

Would be interesting to get a detailed handle on how garbage collection works in PHP and how much of the problem is outstanding bugs vs. fundamental issues with that way PHP does things.

PHP uses reference counting for garbage collection (as do many scripting languages, as it's easy to implement) so there's the fundamental danger of circular references which require "manual" clean up. This is very similar to the type of issues you can have with Javascript in IE (which also uses ref counting I believe).

So assuming you're careful to clean up circular references yourself, in theory PHP shouldn't leak. That said what's been going on with references in PHP recently (which Jeff has now summarized nicely here) may also have been part of the problem - the "memory corruption" that recent fixes have been designed to prevent may also have been a source of "leaks" - memory PHP no longer knows how to free.

Cal Henderson's Flickr (PDF) presentation alluded to garbage collection problems they have - they were using PHP as a process to handle images if I remember right (e.g. resize a 1.2mb image) which is where they encountered problems.

Otherwise a couple of good links on PHP GC here and here.

Hum, it seems like a headeach to me, im not sure how much time will it take to me to learn basic perl, but dont like the idea of writing tests and code in different language, but sure i will consider this further

That's a fair point but long term you may find that using PHP for this is more trouble than it's worth. Need to a slap up a "Perl for PHP programmers" quick start sometime. In general though Perl, PHP, Python (and probably Ruby) are all much of a muchness - once you've grasped one, it's easy to grasp another.

any other ideas?

Probably not the answer you're looking for but Python Network Programming is an excellent book (like I'm saying "right tool for job"). Does a nice job of introducing twisted among other things. Going further off track, what I didn't know about twisted is it uses asynchronous IO (or non-blocking IO) which represents an alternative to using forking or threading. And PHP has stream_set_blocking.

[lastcraft]

Hi.

I had a smaller scale problem (about 1% of the complexity of yours) testing e-mail. See the fakemail page (http://www.lastcraft.com/fakemail.php) for the eventual solution. I fire up a demon and kill it on every test run. I had more than a few socket issues and some help from Pawel (LIMB) before I was done. Linux is a bit fussy about releasing file descriptors. That was Perl, but I would expect the same problems from an external PHP script.

PHP is a lousy testbed for this stuff until Apache/PHP gets proper thread support. Firing up a separate script/process seemed to be the safest to me.

yours, Marcus

[sweatje]

Originally Posted by HarryF

Good point - was actually thinking of other issues like the pcntl extension being relatively untested compared to more well used parts of PHP and that things like OS signals are better integrated in languages like Perl.

Would be interesting to get a detailed handle on how garbage collection works in PHP and how much of the problem is outstanding bugs vs. fundamental issues with that way PHP does things.

Some interesting stuff in this thread. I though Sara Golomon's comments (about half way thought the thread) were interesting in regards to how to track down a suspected memory leak.

[sism82]

It seems to me like i have enough reasons to takea fair decission. I have been reading all the references that all you guys kindly posted. I cannot deny that im a bit "afraid" of what could happend to my daemons, but i guess that with discipline and carefull design it will succeed.
About the testing, i came up with a simple solution that is a mix of what i had and what Marcus made. Not completly decided yet, but the server and client thing has been abstracted into a method, so i can further decide if I fire up two separate scripts with shell commands, or I continue using forks(). Time will tell if forks was not the path to follow. But i guess the PHP developers put a little of effort providing that functions so some dummy guy has to test them, and that would be me -_-

Lets give a review of my testing script:

- First just fireup a proxy client and a proxy server, the proxy client connects to the proxy server. The proxy server connects at the same time with the Asterisk Manager. Then the proxy client proof the connectivity using a 'ping' action. Until that moment, the first true assert is done if the ping response is received sucessfully, the connectivity is OK.

- Second test, client proxy start sending random commands to the Server, the server proxy the actions and responses to/from Asterisk Server. Then i compare the received responses with the expected responses and if are equal, other test passed successfully.

- Third test, i will instance a MockAmiConnector, this is a mock instance of AmiConnector. AmiConnector is the class that helps the Proxy Server to get connected with the AsteriskManager. So i will need to fake this connection in order to send fake Events to the Proxy Server. From that events, only the events wich the proxy client is suscribed to must be sent. So i will compare the sent events against the received ones, and again, a true assert will be issued if everything is ok.

: well, despite the long explanation, the test is simple. Now, what do you think of this: Im testing this system as a whole, because i think that is better ( and simpler ) to test if it works, or not; than testing small pieces in the system isolated. Advantages and disadvantages?? well, i see:

Advantages:
- simple, and simple, and simple
- Does not compromisse quality, since the test will tell us if it works, or not.
- Still is extendable to test more specific parts of the system.

Disadvantages:
- Is not concrete. So when the test does not pass, it wont tell the developers the exact point of failure. It could be at the socket read function, at the argument parsing etc. etc....

Suggestions? critics??

Best Regards.

[lastcraft]

Hi.

I think you are going the right way having a few end to end tests. It would be nice to have unit tests for each part, but what can you do? You acceptance test until you feel confident that the code is working. You unit test as an aid to writing the code, but if you are writing many times more test code than real code you could achieve better quality with some other technique. Code inspection for example.

yours, Marcus

[sism82]

Marcus: thanks for the suggestion about code inspection. Only time will tell me if this is the correct technique to use.

Getting back to my problem. I was able to solve it, but with a workaround for the events tests, that was the
one giving me a headache. Ok, the final code is in here:

http://phpmexic.u33.0web-hosting.com...iProxyTest.php

this is my simple way of showing you the architecture

( Asterisk VoIP server <--> AmiConnector <--> AmiProxy <--> ApSignalHandler a.k.a. Clients... ) <--> AmiProxyTest

ok, first, the logic is always the same

1. The Test class (AmiProxyTest) launch two parallel processes, one of the processes run AmiProxy (the server)
and the other one run the client ApSignalHandler
2. The client process is actually an instance of ApSignalHandler getting as reference for the
callbacks $this (AmiProxyTest). Thats why some methods in AmiProxyTest are callbacks that are called when
the client process receives data.
3. The triggered callbacks write meaninfull data to the defined assert file
4. The parent process waits about 10 seconds or less. After that time the parent kill both child
processes (client and server) and then read the information in the assert file to decide the assertions.

Ok, that is what happend each time a test method is run. Now, the test methods are:

1. testConnectingToManager()
this test sends a ping action from the client, and waits for a pong response from the server.
if the response is received ( the callback method triggered ), the response is written to the
assert file. Then the parent process will check at the end of the test method if the file has
the correct data.

2. testSendingCommands()
some commands are predefined in the test, and its expected responses. the client then sends
all the commands and waits for the respective responses, then the callback methods write the
responses to the file and the parent process will check the responses again.

3. testReceiveAndRelayEvents()
this is the difficult part. Events are things that happen when some user dials a number,
hangup a call, transfers a call, listen to some recording etc. So events should be random,
and cannot be generated from the client (actually we can, but we want to test server events ).
So, i think i had 2 clear options. Create a dummy VoIP server, or just fake the events mocking
the class AmiConnector, that is the one that helps the proxy to communicate with the VoIP
server. So i choosed the option 2. But, sockets are tricky to mock. I could not find an easy way
to trick the socket_select() php function, thus i had a new problem. That was resolved using
partial mocking, so the connection still was done, then i only mocked the method
AmiConnector::ReceiveManagerPackets(). The used some setReturnValueAt() to fake random events.
the random events of course, had fixed data, that the parent process read from the assert file
and checked it for consistency.

ok, now, one thing that just come to my mind to add in the great Marcus simpletest framework is
a setReturnRandomValues() or something, because my workaround is ugly. I just made a loop 100 times
adding the same chain of values for the events. But that will break when the calls to the mocked
method are more than 100 * $number_of_events, after that it will start returning the default value,
and for applications that run forever is usefull to remain returning random values.
well, but that are just my 2 cents. I will check it out, and may be send a patch to marcus, unless
that functionality is already implemented and i missed it.

finally i would like to tell that YES, TDD leads you to find better ways of doing things. Through
the develop of this simple testing suite, i have refactored some things in the architecture in
the proxy that could have lead to failures later.

Regards,

[Etnu]

Originally Posted by sweatje

PHP has notoriously bad memory management. Because the typical usage it to create a PHP process for each request, often freeing memory is delayed until the termination of the process. If you intend to have a long running daemon, this might have a significant impact on you.

unset() usually takes care of it.

I have a billing daemon that runs in PHP. I stress tested it by processing about 50 transactions per second for 8 days straight. Memory usage never climbed over 150MB.

Sure, it's not as good as writing a daemon in, say, C, but it works quite well. PHP's socket implementation is surprisingly good.