Any tips for reducing Apache CPU usage?

My typical top output mid-day…

top - 10:29:31 up 10 days,  8:16,  1 user,  load average: 41.53, 55.29, 46.67
Tasks: 147 total,  18 running, 128 sleeping,   0 stopped,   1 zombie
Cpu(s): 90.2%us,  4.3%sy,  0.0%ni,  1.2%id,  0.0%wa,  0.0%hi,  0.0%si,  4.2%st
Mem:   1788724k total,  1253268k used,   535456k free,   144848k buffers
Swap:   917496k total,        0k used,   917496k free,   271644k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
18821 apache    15   0 60280  21m  13m S  8.7  1.2   0:26.31 httpd
18794 apache    15   0 60504  22m  14m S  8.2  1.3   0:24.76 httpd
18847 apache    15   0 61040  22m  13m S  6.3  1.3   0:18.20 httpd
18945 apache    16   0 60248  20m  12m R  5.3  1.2   0:03.82 httpd
18836 apache    15   0 60804  22m  14m S  5.1  1.3   0:25.08 httpd
18875 apache    15   0 59688  21m  14m R  5.1  1.2   0:18.46 httpd
18948 apache    15   0 60760  20m  12m R  5.1  1.2   0:03.48 httpd
18849 apache    16   0 61040  22m  14m R  4.8  1.3   0:16.71 httpd
18949 apache    15   0 59932  19m  11m S  4.8  1.1   0:02.54 httpd
18830 apache    15   0 60264  21m  13m R  4.6  1.2   0:28.83 httpd
18873 apache    15   0 60788  22m  13m R  4.6  1.3   0:19.59 httpd
18935 apache    15   0 59652  20m  12m R  4.6  1.1   0:08.02 httpd
18941 apache    16   0 59960  19m  11m R  4.6  1.1   0:05.19 httpd
18956 apache    15   0 59964  20m  12m S  4.6  1.2   0:02.89 httpd
18882 apache    15   0 59680  20m  13m S  4.1  1.2   0:10.47 httpd
18959 apache    15   0 60228  20m  12m R  3.9  1.2   0:02.38 httpd
18961 apache    15   0 59696  17m 9948 S  3.9  1.0   0:02.31 httpd
18807 apache    15   0 60796  22m  13m S  3.4  1.3   0:25.94 httpd
18951 apache    15   0 59936  20m  12m S  3.4  1.2   0:02.53 httpd
18824 apache    15   0 59676  21m  13m S  3.1  1.2   0:25.66 httpd
18825 apache    15   0 59680  21m  13m S  3.1  1.2   0:26.07 httpd
18826 apache    16   0 61024  22m  14m R  3.1  1.3   0:24.66 httpd
18885 apache    15   0 60236  21m  13m S  3.1  1.2   0:12.95 httpd
18903 apache    15   0 60248  21m  12m S  3.1  1.2   0:12.05 httpd
18908 apache    15   0 60236  21m  13m S  3.1  1.2   0:12.15 httpd
18911 apache    15   0 60244  20m  12m S  3.1  1.2   0:12.08 httpd
18832 apache    15   0 61048  22m  14m S  2.9  1.3   0:28.84 httpd
18858 apache    15   0 59672  21m  13m S  2.9  1.2   0:19.75 httpd
18924 apache    15   0 59676  19m  12m S  2.9  1.1   0:07.75 httpd
18926 apache    15   0 60204  20m  12m S  2.9  1.1   0:07.40 httpd
18931 apache    15   0 59688  19m  11m S  2.9  1.1   0:06.07 httpd
18954 apache    15   0 59948  19m  11m S  2.9  1.1   0:02.44 httpd
18772 apache    15   0 60628  21m  12m S  2.7  1.2   0:27.89 httpd
18905 apache    15   0 59932  20m  12m S  2.7  1.2   0:12.75 httpd
18906 apache    15   0 60232  21m  13m S  2.7  1.2   0:12.18 httpd
18916 apache    15   0 60240  21m  13m S  2.7  1.2   0:11.39 httpd
18859 apache    15   0 60760  22m  14m S  2.4  1.3   0:21.03 httpd

This is a “High CPU Medium Instance” running on Amazon EC2, so it has the equivalent of at least two dual-core processors. Still taxing the CPU pretty hard as you can see.

This instance is running the web server (and nothing else) for W3Counter. It handles about 40 HTTP requests per second in order to serve up the counter/tracker images on users’ websites and send a few database queries to record the hits.

Any tips for tweaking httpd.conf to reduce the CPU usage further? I have already removed all modules that I don’t need. I’ve disabled hostname lookups. I disabled the access log.

The basic settings:

Timeout 120
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 4

<IfModule prefork.c>
StartServers         10
MinSpareServers      15
MaxSpareServers      50
ServerLimit          512
MaxClients           384
MaxRequestsPerChild  3000
</IfModule>

serve up the counter/tracker images

If the serving of these images is the main function of this server alone, you could consider using a lighter weight server such as lighttpd for the serving of the media, or am I misinterpreting something?

Well, serving up a counter image involves calling stored procedures (MySQLi), generating images (PHP GD), IP-to-location resolution (MaxMind’s GeoIP Apache module), browser identification (PHP’s browscap), and a bunch of other work… and must also be available over SSL.

Will Lighthttpd help if all the HTTP requests involve significant PHP work?

Dan,

About the only thing I know about is that GD is a memory/CPU hog. With that said, it may be worthwhile to store images and merely serve the image based on the returned value from the db. Maybe in sets of 3 for the high volume sites?

I know that PHP also takes a while to do the location resolution but the obvious drawback is maintaining a database of locations by IP. NOT GOOD! On the other hand, how many NEW IP locations are you getting? I’d think that your queries would be reasonably static in terms of IP/location and that would cut down on the time for resolution (if in the db) Just food for thought.

Regards,

DK

Just to see what the effect would be, I disabled mod_geoip for a while and didn’t notice much difference. I’ll try to play around with some image caching to see just how much CPU usage GD causes. Thanks:)

So it’s peak time again, I temporarily disabled mod_geoip and GD rendering. Load didn’t really move.

I think it’s simply as good as apache’s gonna get at handling 40 requests/second, and I need to load balance this over multiple servers :frowning:

So much for trying to consolidate hardware to save on the bills!

Yes, Lighttpd can do that. In almost all cases that I’ve seen a transition from Apache to Lighttpd, the latter scaled much better. Not to mention if I read correctly you’re using Apache Prefork which to my knowledge is not threaded ( worker is ).

You can get PHP and MySQL running no problem, but of course the first time around is a lot harder on a new server. I would recommend using FastCGI for PHP. The GeoIP is still a C API not specific to Apache, you can still manually compile ( see http://www.cyberciti.biz/tips/linux-lighttpd-install-mod_geoip-tutorial.html ).

Honestly there’s a reason the heaviest trafficked sites like Youtube use Lighttpd - it’s just that good. Apache was designed to be bloat by having everything and anything.

So if you have the budget I’d definitely recommend looking into Lighttpd and setting up a test server. If you need help there’s an irc channel devoted to it on Freenode, #lighttpd.

Links:
http://www.lighttpd.net/
irc://irc.freenode.net/#lighttpd

Of course I would also recommend you do your own share of research, best of luck.

I seem to have eliminated the need to reduce the load quite easily.

W3Counter as a site is nicely portable. Check out its Subversion repository on any server and it’s ready to respond to requests. No hard coded paths or URLs and such, no configuration needs to be changed unless the database server moves. It’s even ready to use a partitioned database just by adding the server addresses to a single configuration file (the DB was spread across 3 crappy servers at one point).

Anyway, at the moment, the site runs on two EC2 instances. A “High CPU Medium” instance for the web server (5 EC2 Compute Units, 1.7GB RAM) and a “Large” instance for the database (4 EC2 Compute Units, 7.5GB RAM).

The bulk of the load is a request for the W3Counter javascript on a user’s website, which writes an image tag pointing to the tracking script.

So I checked out a copy of the website on the database server and fired up Apache there.

I added a tiny bit of logic that has the JavaScript write an image tag pointing to the high-CPU instance 60% of the time, and pointing to a URL on the database server 40% of the time.

That little bit of JavaScript load balancing gives the site just enough of a break that the runaway load (was over 100 today) instantly dropped back to single digits… on both servers. The DB normally sat around 2-3 as it’s disk IO bound not CPU bound. Plenty of extra CPU time sitting there.

And no complicated configuration of hardware load balancing, proxies or Amazon Elastic Load Balancers.

Bonus screenshot: Windows Vista gadgets display near-realtime server load, and the anime face changes its expression based on CPU usage.

Jeez… pretty high end servers ya got there… it’s nice you were able to lower the usage, pretty good strategy.

I would still recommend looking into lighttpd if you’re interested in further optimization.

Nice little gadget ya got there too.

You probably need to search which user of your server is using high httpd resource. Currently it will display you username as “Apache” because you might not have enabled “suphp” If you wish then you can enable it and will able to track an exact username running httpd process.

Note: suphp may decrease the server performance.

Pro tip: Read at least one post before replying.

Thanks for drawing my attention. It was replied by mistake.