PHP
Article

Benchmarking: Can AppServer Beat Symfony’s Performance?

By Scott Molinari

Mastering Appserver

After the release of the first part of our Appserver series, it was clear through the ensuing discussions on both SitePoint and Reddit that we had touched a nerve for a good number of PHP channel’s devoted readers. I also quickly realized this new (for PHP) technology had a good number of serious doubters. One of the most poignant responses in the discussions was something along the lines of,

“with the way pthreads works internally with PHP, appserver can never be as fast as a regular LEMP or even LAMP stack.”

The Challenge

Needless to say, those doubtful and critical comments sounded like a real challenge. I was also very interested in finding out where appserver would land, if it were to be benchmarked against another well known PHP framework.

Intro image, Appserver logo versus Symfony logo

I decided to use my favorite framework, Symfony, to make the comparison. This is because appserver, as a stock PHP application server, also offers a good bit of important application functionality similar to Symfony. These functions include a dependency injection container, a request and response object, a method for persistence, etc. In other words, it gives the developer the basics to build an MVC application, just like Symfony does.

For those of you wondering what pthreads is, it is a PHP extension used in the core of appserver, which allows PHP to build a multi-threaded PHP server environment. If you aren’t sure what that means, or why it is at all relevant, please read the previous post.

LEMP or LAMP stacks are the usual way PHP applications are served with Linux.

  • Linux, Nginx (EngineX), MySql and PHP -> LEMP

  • Linux, Apache, MySQL and PHP -> LAMP

The Approach

The basic idea of this post is to try to dispel the criticism about appserver performing poorly, or at least much worse than a framework like Symfony.

The tests were run with simple programs, so we can see the overhead we’d find with standard programs built on top of a larger framework. I consider the overhead typical for any enterprise-grade, professional PHP application.

We will be testing for 3 types of responses.

  1. A straight “Hello World” response. (Hello World page)
  2. A response with no database access, but with dynamic templating (Login page) (Note: to make the test fair, I took out all the content on the login page of the Symfony demo login page, so the document lengths would be similar.
  3. A page with database access and templating (Content page) – Note: to also make this part a fair comparison, I added the content from the first page of Symfony’s blog list to the items list of the appserver demo app.

Comparing it to Symfony was relatively easy, since Symfony can also be installed with a demo application. The appserver demo app isn’t a blog like Symfony’s, but rather a sort of to-do list kind of application. Again, I added the same content from the Symfony Blog posts index page into the “to-do” list index page, so the page sizes were approximately the same.

Although I’ve read one shouldn’t use the Symfony demo for benchmarking, I see no real reason not to in this case. Both “demos” aren’t made for benchmarking really, so they both have the same disadvantages. And, in this little exercise, we want to see how appserver compares in relation to similar kinds of responses, be it dynamic content, hello world or a login page.

Funny enough, they both use Sqlite as the database and doctrine for database abstraction for their demo applications. So, as systems, they are very similar in this respect too.

The Setup

I ran the tests on a 1GB standard VM on Rackspace, so the tests would be as neutral as possible.

The VM was created through Puphpet to build a Debian Wheezy LESP stack (LESP because we are using SQLite). I later installed Symfony and appserver manually. Since appserver’s webserver initially accepts requests through port 9080, it was easy to run both systems in parallel. Symfony is using php-fpm version 5.6.16 and appserver is using its own installed version of php-fpm version 5.6.15. I used PHP5.6 and not PHP7 with Symfony to keep the comparison fair between the two systems and for a proper apples to apples comparison. Appserver only comes with PHP5.6 currently; version 1.2.0 will be arriving with PHP7 some time towards the middle to end of this year.

To test the two systems, I used Apache Bench, similar to the tests done in this PHP framework benchmark repository. I also ran a three second test with 10 concurrent users with only one difference. I used the “-k” keepalive option, which is a standard used for HTTP connections in all web browsers.

The main goal of the tests was to get the number of requests per second.

I did not measure memory usage, as this measurement is not really comparable between Symfony’s standard PHP “shared nothing” and appserver’s threaded environments. That would be an apples to oranges comparison.

The only thing I did for optimization for both systems was carry out the official steps for deploying to a production environment.

The Results

I’ve broken down the results into the three sets of responses, as mentioned above.

Symfony’s Hello World page

Hello World benchmark, Symfony

Symfony’s Login page

Login benchmark, Symfony

Symfony’s Content page

Content benchmark, Symfony

appserver’s Hello World page

Hello World benchmark, Appserver

appserver’s Login Page

Login benchmark, Appserver

appserver’s Content page

Content benchmark, Appserver

Results Graph

Graph showing three bar charts representing requests per second. The first shows that the frameworks are equal on the content page (19 requests per second), the second shows Appserver at 79 and Symfony at 38, and the Hello World experiment shows Symfony at 97 and Appserver at 86

Symfony on appserver

Just to see what would happen, I also installed Symfony onto appserver. Here are the results compared to the LESP (LEMP in the graphic) stack.

Graph showing three bar charts, representing requests per second. The benchmarks were Content page, Login page, Hello World page, in that order. The first, Symfony is 19 vs 15 faster on LEMP. The second, Symfony is 38 vs 28 faster on LEMP. The third, Symfony is 97 vs 57 faster on LEMP.

Conclusion

Looking at the appserver/Symfony vs. LEMP/Symfony results, I’d say the PHP webserver in appserver has additional overhead compared to Nginx. That is pretty much understandable. To be honest though, I would have also guessed a PHP based web server would do a whole lot worse compared to Nginx. So, with these results, I’d give a thumbs up to the dev team for doing a pretty good job with their pure PHP web server implementation.

As one can also see, once a SQLite database gets into the picture, it is certainly the bottleneck. Both slow down considerably when database requests to SQLite come up.

For the login page, appserver wins out. To render the page, it uses dhtml templates. I’d say this might show the overhead Twig puts on Symfony, despite Twig’s caching of the templates.

I would also venture to say that appserver’s slightly poorer hello world result lies within the PHP web server overhead, even though the results are actually quite similar, with Symfony only approximately 16% faster. Compared to Symfony on appserver, the results show appserver’s web server and application server combination do a good job overall in combination, to speed up processing.

I hope this short post will dispel some of the mistrust some people may have about the possible performance one can get out of their applications on appserver. The technology is quite new, but still very interesting, and as we can see, not really all that bad in terms of performance.

An important note: with these tests, we are not testing the real advantages or strengths of appserver. Once an application starts to get much larger, where a larger number of classes/ services needs to be constantly (re)loaded in the standard PHP environment, I believe appserver would start to do considerably better than Symfony. I’ll work on testing this aspect at a later date, as it is actually the real selling point for a threaded PHP environment.

Hopefully, some of you will be motivated to at least give appserver a spin. The results above should dispel any bad performance concerns, and the appserver platform can only improve over time, especially if a good number of developers use it and put it through its paces.

In the next two posts, we’ll start to get our hands dirty and actually develop an appsever application. Stay tuned for more!

  • Jeroen Meeus

    First of, a ‘load test’ of 3 seconds with 60 requests doesn’t really say anything. Especially if there is no CPU/ram consumption data. Run this again and have multiple concurrent users with different session objects hit the server at the same time for 10 minutes + to have a proper comparison.

    More on subject: So basically, you add a dependency on you server that adds the problems a multi threaded environment adds and you have 0 gain when compared to other speed-up solutions (tweaking of apache mpm worker, Caching, HHVM, …)

    Just for information sake, can you also give us your pool.d configuration of fpm?

    • phan7om

      Indeed. Symfony is some kind of slow framework with comparsion to for example Yii

      • http://www.skooppa.com s.molinari

        Yes, you are right. Symfony, in its stock form, isn’t the fastest framework. It is also why I basically intentionally followed the framework benchmarks and also linked to them. The choice of using Symfony was more about my own knowledge more than anything else. That and the goal of comparing appserver to a well known and very professional framework, which Symfony undeniably is.

        Scott

    • http://www.skooppa.com s.molinari

      The php-fpm settings were standard out of the box. The same goes for the worker settings for appserver.

      I am not certain about what you mean with

      ” you add a dependency on you server that adds the problems a multi threaded environment adds and you have 0 gain when compared to other speed-up solutions (tweaking of apache mpm worker, Caching, HHVM, …)”

      If you make a mistake with any system, multi-threaded or not, it will cause you to have problems, when comparing that system to another. The trick is to not make any mistakes and use the advantages a multi-threaded PHP environment has to offer. ;-)

      Scott

  • http://www.skooppa.com s.molinari

    As I sort of mentioned at the end of this article, we aren’t working with a full blown application in this benchmark, so we aren’t really taking any advantage of the extras appserver has to offer, especially its multi-threaded architecture. If we created a full blown large application with appserver and would do the same with Symfony, I bet we could see a noticeable difference in performance.

    Just like the benchmark article for PHPFastCGI states:

    “if the application is doing more there are greater potential time savings to be had”

    Unfortunately, I can’t support this assumption with any facts, as there hasn’t yet been a full blown large application built with appserver, which could also be ran with Symfony (or any other framework for that matter). We won’t ever see this too, as the way appserver works compared to standard PHP applications and frameworks is different.

    There is also the difference to appserver and the other threaded alternatives you mentioned. Appserver moves towards a more full featured application server stack. It doesn’t just try to improve PHP’s lack of shared resources. It goes beyond that. This should be a good thing, one would think. However, exactly this large goal of appserver, to be a full web application stack, could also be its Achilles heel, as it also means devs have to learn and accept more. I guess we’ll have to see how it goes.

    Scott

  • http://waryway.com/ Kyle Wiering

    I find these results interesting but expected. The previous comments kind of elaborate on this without stating it. The comparison here, is explicitly the load balancing and caching on the web server portion. Until there is an application that takes advantage of the application multi-threading aspects, it is going to be hard to beat nginx’s compiled ability to server a ton of stuff fast.

    The real ‘gain’ with something like appserver is the removal of a layer of the onion, and, better integration between the server and the app.

    Additionally, I would bet running appserver on php7 will give it a bigger performance boost relative to running on 5.6 then bumping a lesp stack from 5.6 to 7.

Recommended
Sponsors
Get the latest in PHP, once a week, for free.