SitePoint Sponsor

User Tag List

Results 1 to 11 of 11
  1. #1
    Always A Novice bronze trophy
    K. Wolfe's Avatar
    Join Date
    Nov 2003
    Location
    Columbus, OH
    Posts
    2,182
    Mentioned
    65 Post(s)
    Tagged
    2 Thread(s)

    Anyone have some insight into benchmarking a framework?

    I'm getting ready to do a few benchmarks for a project I'm working on (PHPRouter). I mainly use it for RESTful APIs. I don't expect any crazy results, it will obviously have a little less overhead than a full fledged framework, but I want to put it up against ZF1, ZF2, CI and a few others. I'd like suggestions on other frameworks I should include and also, more importantly, what tool should I use to create these benchmarks? I know Apache has a benchmarking tool that I will be looking in to, should I use this (seems like the default)? I'm mainly after page loads per second.

  2. #2
    SitePoint Wizard bronze trophy Jeff Mott's Avatar
    Join Date
    Jul 2009
    Posts
    1,268
    Mentioned
    18 Post(s)
    Tagged
    0 Thread(s)
    There's an app for t—err, I mean, a Symfony Component for that.

    But if instead you're just interested in the final response time, then yes, the Apache Benchmark (ab) tool is perfectly good for that. Or if you want another choice to consider, then siege is a good tool also.

    But be careful to compare apples with apples. Is your PHPRouter supposed to be a full-stack framework? If not, then it probably doesn't make much sense to compare it against other full-stack frameworks. It may be more useful to compare against just the router component of those frameworks.
    "First make it work. Then make it better."

  3. #3
    Always A Novice bronze trophy
    K. Wolfe's Avatar
    Join Date
    Nov 2003
    Location
    Columbus, OH
    Posts
    2,182
    Mentioned
    65 Post(s)
    Tagged
    2 Thread(s)
    Quote Originally Posted by Jeff Mott View Post
    But be careful to compare apples with apples. Is your PHPRouter supposed to be a full-stack framework? If not, then it probably doesn't make much sense to compare it against other full-stack frameworks. It may be more useful to compare against just the router component of those frameworks.
    It isn't, however I still want to show common frameworks vs PHPRouter vs no routing.

  4. #4
    SitePoint Addict bronze trophy
    Join Date
    Apr 2013
    Location
    Ithaca
    Posts
    351
    Mentioned
    6 Post(s)
    Tagged
    1 Thread(s)
    I am not an expert on this topic, but there's one tiny advice I'd like to make, you better actually build a simple but useful application(maybe an e-commerce site) to run the benchmark tests. Some benchmarking articles use simple 'hello world' as the source for benchmarking test but the problem is, why do you even need PHP at all if all your program does is to print hello world on the screen(I can achieve this with HTML lol)? Some go a little bit further by executing several if conditions, loops or creating objects, even so they may be biased since in real applications you do not just create like say a loop that executes 1000 times that accomplish nothing. To produce a reasonable benchmarking result, you need to be a bit more realistic but creative.

  5. #5
    Always A Novice bronze trophy
    K. Wolfe's Avatar
    Join Date
    Nov 2003
    Location
    Columbus, OH
    Posts
    2,182
    Mentioned
    65 Post(s)
    Tagged
    2 Thread(s)
    Quote Originally Posted by Hall of Famer View Post
    I am not an expert on this topic, but there's one tiny advice I'd like to make, you better actually build a simple but useful application(maybe an e-commerce site) to run the benchmark tests. Some benchmarking articles use simple 'hello world' as the source for benchmarking test but the problem is, why do you even need PHP at all if all your program does is to print hello world on the screen(I can achieve this with HTML lol)? Some go a little bit further by executing several if conditions, loops or creating objects, even so they may be biased since in real applications you do not just create like say a loop that executes 1000 times that accomplish nothing. To produce a reasonable benchmarking result, you need to be a bit more realistic but creative.
    Thanks for your input Hall of Famer. I will actually be using a "hello world" in json. The idea of this is to show the overhead difference between a full framework vs just the router for when you are creating something as simple as a RESTful api. Benchmarking how many requests you can complete in a period of x seconds with a hello world accomplishes quite a lot, actually. It is the easiest way to determine the true overhead of the framework / language. Many do the same (and rightfully so) when comparing web services across languages (go, python, node.js) because certain libraries such as an html or db package might be causing slow downs which can be addressed separately.

  6. #6
    SitePoint Wizard bronze trophy Jeff Mott's Avatar
    Join Date
    Jul 2009
    Posts
    1,268
    Mentioned
    18 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by Hall of Famer View Post
    ...you better actually build a simple but useful application(maybe an e-commerce site) to run the benchmark tests. Some benchmarking articles use simple 'hello world' as the source for benchmarking test but the problem is, why do you even need PHP at all if all your program does is to print hello world on the screen...
    To be continued...
    "First make it work. Then make it better."

  7. #7
    SitePoint Wizard TheRedDevil's Avatar
    Join Date
    Sep 2004
    Location
    Norway
    Posts
    1,196
    Mentioned
    4 Post(s)
    Tagged
    0 Thread(s)
    What you need to remember is that tools like "ab" and "siege" only give a static overview of the possible load.

    The numbers you get from using them normally does not propagate to real world scenarios.

    In your case I don't think this matters, since you just want to "showcase" results vs. other frameworks, so its purely statistic and not for instance configuration.

    Though, for anyone that want to test their new application/website, I would highly recommend one of the services which offer real traffic during the test (most just spawn small instances and then use test cases to do specific things on your system). As this will give you much more valuable information on how much your system can handle at the current server/cluster configuration.

  8. #8
    SitePoint Enthusiast
    Join Date
    Sep 2002
    Location
    European Union
    Posts
    51
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I use Xdebug to create files for Wincachegrind to analyze those files and use it mainly to check for bottlenecks in my applications.

    http://xdebug.org/
    http://sourceforge.net/projects/wincachegrind/

  9. #9
    SitePoint Wizard silver trophybronze trophy asp_funda's Avatar
    Join Date
    Jun 2003
    Location
    ether
    Posts
    4,497
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    You might want to add Slim & Silex in the frameworks you compare against - these 2 are micro frameworks for making very lightweight apps & APIs - the real competition for what you're making rather than full stack frameworks like ZF2, Symfony etc. Also, throw in Phalcon in the pot as well, would be interesting to see if you can beat it.
    Our lives teach us who we are.
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Me - Photo Blog - Personal Blog - Dev Blog
    iG:Syntax Hiliter -- Colourize your code in WordPress!!

  10. #10
    Always A Novice bronze trophy
    K. Wolfe's Avatar
    Join Date
    Nov 2003
    Location
    Columbus, OH
    Posts
    2,182
    Mentioned
    65 Post(s)
    Tagged
    2 Thread(s)
    Quote Originally Posted by asp_funda View Post
    You might want to add Slim & Silex in the frameworks you compare against - these 2 are micro frameworks for making very lightweight apps & APIs - the real competition for what you're making rather than full stack frameworks like ZF2, Symfony etc. Also, throw in Phalcon in the pot as well, would be interesting to see if you can beat it.
    Ah nice. I had stumbled across Slim but wasn't sure of how widely it was used.

  11. #11
    SitePoint Wizard silver trophybronze trophy asp_funda's Avatar
    Join Date
    Jun 2003
    Location
    ether
    Posts
    4,497
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    I've looked into Slim a bit (haven't made any app/API in it yet), haven't looked into Silex. Not sure if there are any other micro frameworks in PHP.
    Our lives teach us who we are.
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Me - Photo Blog - Personal Blog - Dev Blog
    iG:Syntax Hiliter -- Colourize your code in WordPress!!


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •