Crippled VPS

Server Config
1

I recently inherited a niche blog from someone with a mutual interest in the subject matter who was no longer interested in maintaining it.

I have a VPS to host a half dozen small (almost no traffic) websites and for development purposes, and I moved the blog to the VPS. Shortly after moving it to my VPS, I began getting out-of-the-blue out of memory errors and processes crashing left and right. All my sites on that VPS would be up and very responsive and the next second BOOM processes being killed left and right and the websites down and/or incredibly slow.

Long story short(er), I finally traced the problems down to this blog that I had recently moved. Looking through raw logs and error logs, I could see traffic hitting the site as normal, and then all of a sudden a single IP hitting the server hundreds of times in a row (corresponding to the same time the memory errors were triggering).

This was/is happening multiple times/day from different IPs… everything working fine, and then all of a sudden, some random IP just nailing my server hundreds of times over a 20-120 seconds and crippling it.

My initial approach was to start blocking these IPs via htaccess as they came up. The problem with this is that I’m not up to over a hundred IPs added to the list and still adding 4-7/day.

I can’t find any rhyme/reason for the problem, and my host simply wants me to upgrade the memory of my VPS. I’m reluctant to do this though, because the website really doesn’t get an excessive amount of traffic that a VPS shouldn’t be able to handle.

Here is an excerpt from my raw logs from a few weeks ago showing accesses to the website:

173.206.29.247 - - [31/Aug/2010:07:07:50 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:49 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:49 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:51 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:50 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:51 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:51 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:51 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:51 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:51 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:51 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:51 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:52 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:52 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:52 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:52 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:52 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:52 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:52 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:52 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:52 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:53 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:53 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:53 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:54 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:53 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:54 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:54 -0500]
24.199.219.129 - - [31/Aug/2010:07:07:53 -0500]

There are then literally hundreds of accesses from that IP over the next minute with it finally ending at [31/Aug/2010:07:09:00 -0500]… then 10 minutes later, a completely different IP that does the same exact same thing. Then a few hours will go by, and then it will happen again… all in all probably 8-12 times/day.

Other things I’ve thought of:

  • Perhaps the IP I have assigned the website to was a “bad” IP from a previous owner
  • Perhaps the problem is with wordpress or a plugin (though I’ve run several WP blogs, and never had a problem). I have disabled all the plugins except All-in-one SEO.

Sorry for the lengthy post, but I thought it better to give more detail than less. At my wits end with this problem here, as my VPS is still crashing at least 2-3 times/day and I can’t figure out how to remedy the problem.

Any ideas?

2

One thing you haven’t mentioned is whether the requests were for legitimate URL’s. The pattern you describe would match the random botnet scans most servers receive looking for a list of vulnerable or misconfigured common web applications. In these cases, they’d get a load of 404 responses which shouldn’t tie up resources sufficiently to affect the server though.

3

If it’s just a few IPs hitting the server hard, try blocking them

4

You’ve done a lookup on the ip and checked it’s not a legitimate spider?
Mod_evasive for apache would certainly deal with this sort of thing if not legitimate traffic.

It’s a lot more likely the multiple hits are causing OOMS, which may indicate you haven’t correctly configured apache which is being allowed to spawn too many processes for the amount of memory you have, and is maxing the memory out.

5

Since you have a VPS, I suspect you have access to IPTables. Using iptables or even a .htaccess, you can just block access to that IP.

If it is only one IP, it could be something mis-configured, somebody targeting this forum or something related to the plugin you have installed.

I would block the IP and see if the site functions ok. If yes, I would not worry about this too much. I’ve seen mis-configured servers hammer domains. You just have to drop the traffic.

If the problems persist, then you may have a plugin causing issues. I’ve worked on some busy wordpress sites and have almost always tracked performance issues to a plugin.

6

Having been through the ECCouncil’s CEH course (http://www.eccouncil.org/certification/certified_ethical_hacker.aspx), I’d say that it sounds like hackers are performing a DOS attack on your server - likely, by the timing, because of the WP forum. Can you confirm it’s just the http daemon (port 80) that’s being hammered or is it happening on other ports, too?

If this IS the cause, you can be protected by (1) keeping your software up to date and (2) implementing programs to prevent the DOS. Saying that, I believe it’s your host’s responsibility to stop this sort of attack - at least if you’re on a managed server. Start by asking them to check the logs on all your server’s ports then hammer them for not detecting or responding to the attacks!

Regards,

DK

7

Yes, I look up each IP to make sure I’m not blocking something like google, or something else. Most of the IPs are just random it seems though. For instance, I had one today from a technical college in Salt Lake City, then about 30 minutes later, one from the Czech Republic.

I’ll look into mod_evasive, thanks for the tip.

8

Thanks for the response.

  • I haven’t noticed a trend of the crashes happening at the same time, but I’ll check that, thanks.
  • No cron jobs are functioning on this website.
  • Regarding suspicious activity like IPs hitting many pages in a short time, that’s basically the scenario in a nutshell and what I’m trying to figure out.

I have disabled the plugin a few weeks ago and monitored the website for a few days and continued to see the weird activity, so I ruled out the plugin as the source and reenabled it.

Also, I have a chicken/egg issue here that I can’t figure out. Are the OOM errors happening first causing some random user out there’s browser to continually ping the server (generating all the hits I’m seeing in my error reports), or is all the hits to the server generating the OOM errors?

I can’t figure out which is coming first.

9

Is Wordpress up to date? Try disabling the All-in-one SEO plugin, see if the problems stop.

  • Are the crashes always at the same time each day?
  • Are any cron jobs scheduled to run around the same time each day that it’s crashing
  • Any suspicious user activity, like IPs hitting many pages in a short time?