Benchmarking: Can AppServer Beat Symfony’s Performance?

After the release of the first part of our Appserver series, it was clear through the ensuing discussions on both SitePoint and Reddit that we had touched a nerve for a good number of PHP channel’s devoted readers. I also quickly realized this new (for PHP) technology had a good number of serious doubters. One of the most poignant responses in the discussions was something along the lines of,

“with the way pthreads works internally with PHP, appserver can never be as fast as a regular LEMP or even LAMP stack.”

The Challenge

Needless to say, those doubtful and critical comments sounded like a real challenge. I was also very interested in finding out where appserver would land, if it were to be benchmarked against another well known PHP framework.

I decided to use my favorite framework, Symfony, to make the comparison. This is because appserver, as a stock PHP application server, also offers a good bit of important application functionality similar to Symfony. These functions include a dependency injection container, a request and response object, a method for persistence, etc. In other words, it gives the developer the basics to build an MVC application, just like Symfony does.

For those of you wondering what pthreads is, it is a PHP extension used in the core of appserver, which allows PHP to build a multi-threaded PHP server environment. If you aren’t sure what that means, or why it is at all relevant, please read the previous post.

LEMP or LAMP stacks are the usual way PHP applications are served with Linux.

• Linux, Nginx (EngineX), MySql and PHP -> LEMP

• Linux, Apache, MySQL and PHP -> LAMP

The Approach

The basic idea of this post is to try to dispel the criticism about appserver performing poorly, or at least much worse than a framework like Symfony.

The tests were run with simple programs, so we can see the overhead we’d find with standard programs built on top of a larger framework. I consider the overhead typical for any enterprise-grade, professional PHP application.

We will be testing for 3 types of responses.

1. A straight “Hello World” response. (Hello World page)
2. A response with no database access, but with dynamic templating (Login page) (Note: to make the test fair, I took out all the content on the login page of the Symfony demo login page, so the document lengths would be similar.
3. A page with database access and templating (Content page) – Note: to also make this part a fair comparison, I added the content from the first page of Symfony’s blog list to the items list of the appserver demo app.

Comparing it to Symfony was relatively easy, since Symfony can also be installed with a demo application. The appserver demo app isn’t a blog like Symfony’s, but rather a sort of to-do list kind of application. Again, I added the same content from the Symfony Blog posts index page into the “to-do” list index page, so the page sizes were approximately the same.

Although I’ve read one shouldn’t use the Symfony demo for benchmarking, I see no real reason not to in this case. Both “demos” aren’t made for benchmarking really, so they both have the same disadvantages. And, in this little exercise, we want to see how appserver compares in relation to similar kinds of responses, be it dynamic content, hello world or a login page.

Funny enough, they both use Sqlite as the database and doctrine for database abstraction for their demo applications. So, as systems, they are very similar in this respect too.

The Setup

I ran the tests on a 1GB standard VM on Rackspace, so the tests would be as neutral as possible.

The VM was created through Puphpet to build a Debian Wheezy LESP stack (LESP because we are using SQLite). I later installed Symfony and appserver manually. Since appserver’s webserver initially accepts requests through port 9080, it was easy to run both systems in parallel. Symfony is using php-fpm version 5.6.16 and appserver is using its own installed version of php-fpm version 5.6.15. I used PHP5.6 and not PHP7 with Symfony to keep the comparison fair between the two systems and for a proper apples to apples comparison. Appserver only comes with PHP5.6 currently; version 1.2.0 will be arriving with PHP7 some time towards the middle to end of this year.

To test the two systems, I used Apache Bench, similar to the tests done in this PHP framework benchmark repository. I also ran a three second test with 10 concurrent users with only one difference. I used the “-k” keepalive option, which is a standard used for HTTP connections in all web browsers.

The main goal of the tests was to get the number of requests per second.

I did not measure memory usage, as this measurement is not really comparable between Symfony’s standard PHP “shared nothing” and appserver’s threaded environments. That would be an apples to oranges comparison.

The only thing I did for optimization for both systems was carry out the official steps for deploying to a production environment.

The Results

I’ve broken down the results into the three sets of responses, as mentioned above.

Symfony’s Hello World page

Symfony’s Login page

Symfony’s Content page

appserver’s Hello World page

appserver’s Login Page

appserver’s Content page

Results Graph

Symfony on appserver

Just to see what would happen, I also installed Symfony onto appserver. Here are the results compared to the LESP (LEMP in the graphic) stack.

Conclusion

Looking at the appserver/Symfony vs. LEMP/Symfony results, I’d say the PHP webserver in appserver has additional overhead compared to Nginx. That is pretty much understandable. To be honest though, I would have also guessed a PHP based web server would do a whole lot worse compared to Nginx. So, with these results, I’d give a thumbs up to the dev team for doing a pretty good job with their pure PHP web server implementation.

As one can also see, once a SQLite database gets into the picture, it is certainly the bottleneck. Both slow down considerably when database requests to SQLite come up.

For the login page, appserver wins out. To render the page, it uses dhtml templates. I’d say this might show the overhead Twig puts on Symfony, despite Twig’s caching of the templates.

I would also venture to say that appserver’s slightly poorer hello world result lies within the PHP web server overhead, even though the results are actually quite similar, with Symfony only approximately 16% faster. Compared to Symfony on appserver, the results show appserver’s web server and application server combination do a good job overall in combination, to speed up processing.

I hope this short post will dispel some of the mistrust some people may have about the possible performance one can get out of their applications on appserver. The technology is quite new, but still very interesting, and as we can see, not really all that bad in terms of performance.

An important note: with these tests, we are not testing the real advantages or strengths of appserver. Once an application starts to get much larger, where a larger number of classes/ services needs to be constantly (re)loaded in the standard PHP environment, I believe appserver would start to do considerably better than Symfony. I’ll work on testing this aspect at a later date, as it is actually the real selling point for a threaded PHP environment.

Hopefully, some of you will be motivated to at least give appserver a spin. The results above should dispel any bad performance concerns, and the appserver platform can only improve over time, especially if a good number of developers use it and put it through its paces.

In the next two posts, we’ll start to get our hands dirty and actually develop an appsever application. Stay tuned for more!

Frequently Asked Questions on Symfony’s Performance and Benchmarking