counting number of hits per second at run time

[pintu_asim]

Hi All,
I am relatively new to PHP.
I need to develope a script that will count number of the hits it gets per second.
I also need to show it in graphs.
ALL THESE HAS TO BE DONE AT RUNTIME.
Any idea how to do it?

Thanks in advance..
Asim

[Austin_Hastings]

Use MySql's Unix timestamp field to create/increment a counter corresponding to the current second.

Compute instantaneous hits/second by fetching the counter.

Compute average hits/second over a time period by using SELECT AVG(count) FROM table WHERE timestamp >= begin AND timestamp <= end

Draw graphs using the GD library to generate a graphical image (it supports several flavors).

=Austin

[pintu_asim]

Thanks Austin,
The proposed script may experience a heavy load up to 100 hits per second.
Please correct me if I am wrong but I think it will not be feasible to do database activity for each hit, as it will create much overhead.
So I think it has to be done in the script itself and after each second it can be stored in the database.
So please propose something so that I can do it in the script level itself.

Thanks again.
Asim

[pintu_asim]

Thanks Austin,
The proposed script may experience a heavy load up to 100 hits per second.
Please correct me if I am wrong but I think it will not be feasible to do database activity for each hit, as it will create much overhead.
So I think it has to be done in the script itself and after each second it can be stored in the database.
So please propose something so that I can do it in the script level itself.

Thanks again.
Asim

[noddy]

Logging such information to a text file will use much more respurces and take a much longer period of time to compute than a mysql database will to store and compile the same information.

[pintu_asim]

hi noddy,
thanks for the reply.
i have some doubts..
if the script is getting around 200 hits/second or more will it be able to keep the count of the hits/second properly at run time? is php capable of doing such things?
according to the proposed solution, for each hit i should read the corresponding record and then update it.will it be feasible to do such operations 200 times per second (or more) ?
one more question..
should synchronization be implemented for this purpose or can i do it safely without going for synchronization?

-Asim

[noddy]

When you try and update a single text file at 200 connections per second you have to open lock the file write to it then close the connection. in order to do this 200 times a second you are going to need a very very fast cpu with plenty of ram.

a mysql database is built on the basis that multiple if not hundreds of transactions and instances will be run simultaneously

See this link for more info http://dev.mysql.com/doc/

Can I ask what site this is going to be for? 200 connections or visitors is a lot and I manage a site that does an average of 1 -1.6gig of bandwidth on a daily basis.

[pintu_asim]

hi noddy,
actually i'm relatively new to php and i came to know that php is more than just dynamic web page designing. so i was trying to do this stuff using php.
this task is not for any site..actually the script will be hit by any load generator and i have to keep track of the hits so that it can judge whether the load generator is generating the proper load or not..
is php capable of doing such things??
can you propose some solution for this??
thanks for your co-operation..
waiting for your next response.

-Asim

[noddy]

Are you talking about the server or computers load indicator on a linux based system?

eg Load Averages 0.00 0.00 0.00

[pintu_asim]

ya something like that.
the script will be hit directly by the load generators @ 100/200 hits/second.
and it has to count the correct number of hits it gets.
i have developed something. what i am doing is i have maintained 2 files namely time.count and hits.count.
each time a hit comes, it compares the current timestamp with the time.count value and if they are same, increments the hits.count value by 1.
when the current time increases than the time.count value i.e. next second, it writes the value in hits.count into database and resets the values of hits.count to 1.also it resets the value of time.count as the current time.
this works fine if i'm giving hits from browsers..around 20 hits/second (constatntly refreshing). but if a load generator is generating say 100/200 threads per second and hits it, it fails badly.
i need some help regrading this.
waiting for your response.

-Asim

[pintu_asim]

hi,
the latest developement is i'm now doin' database operation for each hit and it is working fine upto 150 hits/second.
is there any better way to accomplish the task.how can i improve the performance??

-Asim

[Austin_Hastings]

hi,

how can i improve the performance??

What are your performance objectives?

One way to improve things would be to use an in-memory table in mysql.

Another way would be to use shared memory in PHP.

Another would be to get a faster server.

But first you specified "up to 100 hits per second." Then it was "200 hits per second." <Advisor edit>Watch your language. -Helge</Advisor edit>Do you have some real objective in mind?

=Austin

[pintu_asim]

Hi,
As I mentioned in the previous posts, I'm trying to generate a tool that will calculate the no of hits/second at runtime.
I require it to judge the accuracy of the load generators.
Thus it is obvious that the load may be anything, it may be 100 hits/second or 200 hits/second or more.
Now I’m successful in showing 150 hits/second and trying to improve it further. That is the objective.
Thanks.

-Asim

[PeterW]

another idea might be to just write an apache module which you can connect to with telnet or something

[Austin_Hastings]

Hi,

I require it to judge the accuracy of the load generators.

So the tool doesn't have to run in real time.

Why not just use the access_log of your Apache server (there are corresponding logs for other servers) that contains a timestamp for all requests. Mine looks something like this:

  192.6.6.6 - - [10/Feb/2005:01:54:56 -0500] "GET /twiki/bin/view/Info-you-dont-need;contenttype=text/css HTTP/1.1" 400 299

Conspicuously, the timestamp is surrounded by [], and contains the time including seconds.

In this scenario, you can write a script to return some random gobbledegook, and then at the end of the run (or every 15 minutes, say) just process the log file counting hits per second.

  grep GET access_log \
  | cut -d']' -f1 \
  | cut -d'[' -f2 \
  | uniq -c

=Austin

[Icheb]

More importantly, what kind of MySQL query are you using? You mentioned something like selecting the value and then updating it, which is not recommended. You should directly update the value by using UPDATE table SET hits=hits+1 WHERE bla=bla .

[pintu_asim]

Hi,
Reading the apache access log file is a great idea.
Initially I thought of doing that, but later on dropped the idea because the requirement was to show the graph at run time.
Still I will be trying to implement the same some time in near future.

Currently what I am doing is, just inserting one record per each hit (the timestamp).
Then another PHP script reads the mysql database, gets the no. of entries for each second and plots the graph accordingly. The same is done at 1-second delay (at 10:56:05 the graph is plotted for 10:56:04)

At the same time the cron job runs a script to delete the old records from the table in one minute.

I have some queries..
1. Does the insert statement take more time to be executed than the update statement? (I used insert statement because I thought it would be faster than update)
2. For each hit, I'm making the database connection, inserting the record and closing the connection. Is there any better method to the task?

Many thanks for your kind co-operation.

-Asim

[pintu_asim]

Hi,
Reading the apache access log file is a great idea.
Initially I thought of doing that, but later on dropped the idea because the requirement was to show the graph at run time.
Still I will be trying to implement the same some time in near future.

Currently what I am doing is, just inserting one record per each hit (the timestamp).
Then another PHP script reads the mysql database, gets the no. of entries for each second and plots the graph accordingly. The same is done at 1-second delay (at 10:56:05 the graph is plotted for 10:56:04)

At the same time the cron job runs a script to delete the old records from the table in one minute.

I have some queries..
1. Does the insert statement take more time to be executed than the update statement? (I used insert statement because I thought it would be faster than update)
2. For each hit, I'm making the database connection, inserting the record and closing the connection. Is there any better method to the task?

Many thanks for your kind co-operation.

-Asim

[toggg]

Hi,
Obviously doing that thru some SQL or files will auto limit the max number of hits / second.
It could even be that the "counting/reporting" costs more as the script to study itself.
Everything should stay in memory, no disks.

PeterW idea of Apache module could certainly be the most efficient: Apache is in a central position there and do know the page it serves OK but how develop a module for apache ? BTW it's a good chance that allready exists.

I would try thru a socket where script under observation put a timestamp/ID and from which the analyzing script get the data.
Sure you can observe 10/100 times quicker.

OK, it's a great job to develop, but I could be interested too.
Should I go ahead, try to see how it can be done ?
à+

[pintu_asim]

Hi,
Your concept sounds different and interesting.
It would be so kind of you if you do something in this regard.
Thanks in advance.

Asim

[pintu_asim]

Hi,
Your concept sounds different and interesting.
It would be so kind of you if you do something in this regard.
Thanks in advance.

Asim

[toggg]

OK,
Could you post the script you have at present ?
No matter it works or not
So I can reproduce the concept with accuracy.
à+

[toggg]

He Asim, tell me why you allways post your messages twice ?
Keep cool man, time is no key (for human beings, not for scripts
à+

[pintu_asim]

hi,
the script is something like this..
simple..few lines long

<?php

$hostname_mySQL = "10.8.144.133";
$database_mySQL = "counter";
$username_mySQL = "root";
$password_mySQL = "root";
$mySQL = mysql_pconnect($hostname_mySQL, $username_mySQL, $password_mySQL) or die(mysql_error());

mysql_select_db($database_mySQL, $mySQL) or die("unable to select database 'db': " . mysql_error());
$sql = "insert into counter values('".time()."')";
$fields = mysql_query($sql);
if (!$fields) {
die("query failed: " . mysql_error());
}
mysql_close($mySQL);
?>

A load generator hits the above script at any no. of hits/second.
Intially I tried using files, but that was not so successful.
This script works upto some extent.

There is another script also which takes the value (count(*) for each second i.e. timestamp) from the database and plots a graph..That works fine.

Ya, actually I'm posting it twice to keep it always on top
I'll post it once from now on. Thanks

-Asim

[toggg]

Hi,

Ya, actually I'm posting it twice to keep it always on top

You're right to do that, but pehaps wait an hour or two before
OK for the poster script.
Could you also show the other side of the pipe, I mean this plotter, or at least how it reads and cumulate the data.
I like that, go on, we make it 100 times quicker.
à+