Normal load only have 200+ tasks, but sometime suddenly get High load with 500+ tasks

Server Config
1

Guys i am despert here, already get this problem for a week sometime suddenly get high load, and still dont find where is the problem yet

normally the top result like this, load only 0.71, only have 200+ tasks

other screenshots : http://i291.photobucket.com/albums/ll284/basketmen/topnormal2.jpg

but sometime suddenly get high load (usually 1-3 times a day), load very high 90-200, with 500+ tasks

other screenshots : http://i291.photobucket.com/albums/ll284/basketmen/tophighload2.jpg

Daily Process Log, i already repair and optimize all table, like you can see the mysql process is very low (in white line, not in yellow or red line), so its looks like the problem is not in mysql
The red line is maybe apache

Nb1. I use a dedicated server, only host 1 site, here is my server specs
Intel Single Xeon E3110 3.0ghz L2 cache 6MB
Memory 4 GB DDR2 RAM
Harddisk 500 GB Sata 7200rpm
Cent OS 5.4
WHM/Cpanel

Nb2. I already check no backup/cronjob or related scheduled active right now

Nb3. I already disabled the feature one by one, Even i already block all search engine spider now using robots.txt disallow for 2 days, no spiders again in my website until the problem found, its ok i just need to found the source problem

Nb4. I dont think my hosting support can help it, and they cost very expensive for that extra managed service ($69/hour), i want to find the source problem by my self, please analyze and give advice to found the source problem

2

At a guess you have a misconfigured apache configuration that is allowing too many children to spawn, which is then causing the heavy swap usage you’re seeing in the second pic, which in turn is causing the high cpu wait load as stuff is swapped to disk (not helped by the fact you have a slow disk too). Your mysqld process is using a lot of memory too under load. Try reducing the maximum amount of apache children to what will fit within memory (150 would be a sensible number for you) and check your mysql use for inefficient queries.

3

You’re exhausting your RAM and hitting swap, which will slow you right down to the point where Apache will start giving up a fair bit. In fact you are even exhausting swap too, meaning that you’ll be having out of memory errors: your processes will be being randomly killed!

MySQL’s not using much memory. It looks like it’s all your apache processes, but I can’t be sure without seeing the output of “ps -AH v”

To reduce the maximum number of Apache child processes (as EastCoast suggested) you use the MaxClients directive.

This will be in your httpd.conf.

You have 4GB RAM and most of your Apache children are around 5MB. A back of the envelope style calculation suggestions you should be able to have a huge value for MaxClients, like 400! So I have no idea why you have the problem. But you could try reduce it to less, say 50, and see if the problem remains. Even less perhaps. No shame in MaxClients below 20 - tasks will still queue up nicely.

Note that I use apache2 and you’re on httpd (apache 1.x I presume) - hopefully it’s still called MaxClients but I dunno.