SitePoint Sponsor

User Tag List

Page 1 of 3 123 LastLast
Results 1 to 25 of 59
  1. #1
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Question counting number of hits per second at run time

    Hi All,
    I am relatively new to PHP.
    I need to develope a script that will count number of the hits it gets per second.
    I also need to show it in graphs.
    ALL THESE HAS TO BE DONE AT RUNTIME.
    Any idea how to do it?

    Thanks in advance..
    Asim

  2. #2
    SitePoint Evangelist
    Join Date
    May 2004
    Location
    New Jersey, USA
    Posts
    567
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Use MySql's Unix timestamp field to create/increment a counter corresponding to the current second.

    Compute instantaneous hits/second by fetching the counter.

    Compute average hits/second over a time period by using SELECT AVG(count) FROM table WHERE timestamp >= begin AND timestamp <= end

    Draw graphs using the GD library to generate a graphical image (it supports several flavors).

    =Austin

  3. #3
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks Austin,
    The proposed script may experience a heavy load up to 100 hits per second.
    Please correct me if I am wrong but I think it will not be feasible to do database activity for each hit, as it will create much overhead.
    So I think it has to be done in the script itself and after each second it can be stored in the database.
    So please propose something so that I can do it in the script level itself.

    Thanks again.
    Asim

  4. #4
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Thanks Austin,
    The proposed script may experience a heavy load up to 100 hits per second.
    Please correct me if I am wrong but I think it will not be feasible to do database activity for each hit, as it will create much overhead.
    So I think it has to be done in the script itself and after each second it can be stored in the database.
    So please propose something so that I can do it in the script level itself.

    Thanks again.
    Asim

  5. #5
    Fully Sweet Car noddy's Avatar
    Join Date
    Aug 2002
    Location
    Perth, Western Australia
    Posts
    759
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Logging such information to a text file will use much more respurces and take a much longer period of time to compute than a mysql database will to store and compile the same information.

  6. #6
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    hi noddy,
    thanks for the reply.
    i have some doubts..
    if the script is getting around 200 hits/second or more will it be able to keep the count of the hits/second properly at run time? is php capable of doing such things?
    according to the proposed solution, for each hit i should read the corresponding record and then update it.will it be feasible to do such operations 200 times per second (or more) ?
    one more question..
    should synchronization be implemented for this purpose or can i do it safely without going for synchronization?

    -Asim

  7. #7
    Fully Sweet Car noddy's Avatar
    Join Date
    Aug 2002
    Location
    Perth, Western Australia
    Posts
    759
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    When you try and update a single text file at 200 connections per second you have to open lock the file write to it then close the connection. in order to do this 200 times a second you are going to need a very very fast cpu with plenty of ram.

    a mysql database is built on the basis that multiple if not hundreds of transactions and instances will be run simultaneously

    See this link for more info http://dev.mysql.com/doc/

    Can I ask what site this is going to be for? 200 connections or visitors is a lot and I manage a site that does an average of 1 -1.6gig of bandwidth on a daily basis.

  8. #8
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    hi noddy,
    actually i'm relatively new to php and i came to know that php is more than just dynamic web page designing. so i was trying to do this stuff using php.
    this task is not for any site..actually the script will be hit by any load generator and i have to keep track of the hits so that it can judge whether the load generator is generating the proper load or not..
    is php capable of doing such things??
    can you propose some solution for this??
    thanks for your co-operation..
    waiting for your next response.

    -Asim

  9. #9
    Fully Sweet Car noddy's Avatar
    Join Date
    Aug 2002
    Location
    Perth, Western Australia
    Posts
    759
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Are you talking about the server or computers load indicator on a linux based system?

    eg Load Averages 0.00 0.00 0.00

  10. #10
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    ya something like that.
    the script will be hit directly by the load generators @ 100/200 hits/second.
    and it has to count the correct number of hits it gets.
    i have developed something. what i am doing is i have maintained 2 files namely time.count and hits.count.
    each time a hit comes, it compares the current timestamp with the time.count value and if they are same, increments the hits.count value by 1.
    when the current time increases than the time.count value i.e. next second, it writes the value in hits.count into database and resets the values of hits.count to 1.also it resets the value of time.count as the current time.
    this works fine if i'm giving hits from browsers..around 20 hits/second (constatntly refreshing). but if a load generator is generating say 100/200 threads per second and hits it, it fails badly.
    i need some help regrading this.
    waiting for your response.

    -Asim

  11. #11
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    hi,
    the latest developement is i'm now doin' database operation for each hit and it is working fine upto 150 hits/second.
    is there any better way to accomplish the task.how can i improve the performance??

    -Asim

  12. #12
    SitePoint Evangelist
    Join Date
    May 2004
    Location
    New Jersey, USA
    Posts
    567
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by pintu_asim
    hi,

    how can i improve the performance??
    What are your performance objectives?

    One way to improve things would be to use an in-memory table in mysql.

    Another way would be to use shared memory in PHP.

    Another would be to get a faster server.

    But first you specified "up to 100 hits per second." Then it was "200 hits per second." <Advisor edit>Watch your language. -Helge</Advisor edit>Do you have some real objective in mind?

    =Austin
    Last edited by Helge; Feb 17, 2005 at 05:46. Reason: Bad language. -Helge

  13. #13
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi,
    As I mentioned in the previous posts, I'm trying to generate a tool that will calculate the no of hits/second at runtime.
    I require it to judge the accuracy of the load generators.
    Thus it is obvious that the load may be anything, it may be 100 hits/second or 200 hits/second or more.
    Now Im successful in showing 150 hits/second and trying to improve it further. That is the objective.
    Thanks.

    -Asim

  14. #14
    SitePoint Evangelist
    Join Date
    Jul 2000
    Location
    Warwickshire, England
    Posts
    557
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    another idea might be to just write an apache module which you can connect to with telnet or something

  15. #15
    SitePoint Evangelist
    Join Date
    May 2004
    Location
    New Jersey, USA
    Posts
    567
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Quote Originally Posted by pintu_asim
    Hi,

    I require it to judge the accuracy of the load generators.
    So the tool doesn't have to run in real time.

    Why not just use the access_log of your Apache server (there are corresponding logs for other servers) that contains a timestamp for all requests. Mine looks something like this:

    Code:
      192.6.6.6 - - [10/Feb/2005:01:54:56 -0500] "GET /twiki/bin/view/Info-you-dont-need;contenttype=text/css HTTP/1.1" 400 299
    Conspicuously, the timestamp is surrounded by [], and contains the time including seconds.

    In this scenario, you can write a script to return some random gobbledegook, and then at the end of the run (or every 15 minutes, say) just process the log file counting hits per second.

    Code:
      grep GET access_log \
      | cut -d']' -f1 \
      | cut -d'[' -f2 \
      | uniq -c
    =Austin

  16. #16
    Non-Member Icheb's Avatar
    Join Date
    Mar 2003
    Location
    Germany
    Posts
    1,474
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    More importantly, what kind of MySQL query are you using? You mentioned something like selecting the value and then updating it, which is not recommended. You should directly update the value by using UPDATE table SET hits=hits+1 WHERE bla=bla .

  17. #17
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi,
    Reading the apache access log file is a great idea.
    Initially I thought of doing that, but later on dropped the idea because the requirement was to show the graph at run time.
    Still I will be trying to implement the same some time in near future.

    Currently what I am doing is, just inserting one record per each hit (the timestamp).
    Then another PHP script reads the mysql database, gets the no. of entries for each second and plots the graph accordingly. The same is done at 1-second delay (at 10:56:05 the graph is plotted for 10:56:04)

    At the same time the cron job runs a script to delete the old records from the table in one minute.

    I have some queries..
    1. Does the insert statement take more time to be executed than the update statement? (I used insert statement because I thought it would be faster than update)
    2. For each hit, I'm making the database connection, inserting the record and closing the connection. Is there any better method to the task?

    Many thanks for your kind co-operation.

    -Asim

  18. #18
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi,
    Reading the apache access log file is a great idea.
    Initially I thought of doing that, but later on dropped the idea because the requirement was to show the graph at run time.
    Still I will be trying to implement the same some time in near future.

    Currently what I am doing is, just inserting one record per each hit (the timestamp).
    Then another PHP script reads the mysql database, gets the no. of entries for each second and plots the graph accordingly. The same is done at 1-second delay (at 10:56:05 the graph is plotted for 10:56:04)

    At the same time the cron job runs a script to delete the old records from the table in one minute.

    I have some queries..
    1. Does the insert statement take more time to be executed than the update statement? (I used insert statement because I thought it would be faster than update)
    2. For each hit, I'm making the database connection, inserting the record and closing the connection. Is there any better method to the task?

    Many thanks for your kind co-operation.

    -Asim

  19. #19
    SitePoint Addict toggg's Avatar
    Join Date
    Jan 2005
    Location
    Auvergne/France
    Posts
    253
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi,
    Obviously doing that thru some SQL or files will auto limit the max number of hits / second.
    It could even be that the "counting/reporting" costs more as the script to study itself.
    Everything should stay in memory, no disks.

    PeterW idea of Apache module could certainly be the most efficient: Apache is in a central position there and do know the page it serves OK but how develop a module for apache ? BTW it's a good chance that allready exists.

    I would try thru a socket where script under observation put a timestamp/ID and from which the analyzing script get the data.
    Sure you can observe 10/100 times quicker.

    OK, it's a great job to develop, but I could be interested too.
    Should I go ahead, try to see how it can be done ?
    +
    bertrand Gugger toggg.com linux, PHP, Auvergne/France open source

  20. #20
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi,
    Your concept sounds different and interesting.
    It would be so kind of you if you do something in this regard.
    Thanks in advance.

    Asim

  21. #21
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi,
    Your concept sounds different and interesting.
    It would be so kind of you if you do something in this regard.
    Thanks in advance.

    Asim

  22. #22
    SitePoint Addict toggg's Avatar
    Join Date
    Jan 2005
    Location
    Auvergne/France
    Posts
    253
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    OK,
    Could you post the script you have at present ?
    No matter it works or not
    So I can reproduce the concept with accuracy.
    +
    bertrand Gugger toggg.com linux, PHP, Auvergne/France open source

  23. #23
    SitePoint Addict toggg's Avatar
    Join Date
    Jan 2005
    Location
    Auvergne/France
    Posts
    253
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    He Asim, tell me why you allways post your messages twice ?
    Keep cool man, time is no key (for human beings, not for scripts
    +
    bertrand Gugger toggg.com linux, PHP, Auvergne/France open source

  24. #24
    SitePoint Enthusiast
    Join Date
    Feb 2005
    Posts
    38
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    hi,
    the script is something like this..
    simple..few lines long

    <?php

    $hostname_mySQL = "10.8.144.133";
    $database_mySQL = "counter";
    $username_mySQL = "root";
    $password_mySQL = "root";
    $mySQL = mysql_pconnect($hostname_mySQL, $username_mySQL, $password_mySQL) or die(mysql_error());

    mysql_select_db($database_mySQL, $mySQL) or die("unable to select database 'db': " . mysql_error());
    $sql = "insert into counter values('".time()."')";
    $fields = mysql_query($sql);
    if (!$fields) {
    die("query failed: " . mysql_error());
    }
    mysql_close($mySQL);
    ?>

    A load generator hits the above script at any no. of hits/second.
    Intially I tried using files, but that was not so successful.
    This script works upto some extent.

    There is another script also which takes the value (count(*) for each second i.e. timestamp) from the database and plots a graph..That works fine.

    Ya, actually I'm posting it twice to keep it always on top
    I'll post it once from now on. Thanks

    -Asim

  25. #25
    SitePoint Addict toggg's Avatar
    Join Date
    Jan 2005
    Location
    Auvergne/France
    Posts
    253
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hi,
    Ya, actually I'm posting it twice to keep it always on top
    You're right to do that, but pehaps wait an hour or two before
    OK for the poster script.
    Could you also show the other side of the pipe, I mean this plotter, or at least how it reads and cumulate the data.
    I like that, go on, we make it 100 times quicker.
    +
    bertrand Gugger toggg.com linux, PHP, Auvergne/France open source


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •