After fought against bandwith usage with Baidu I am now fighting against an “Unknown bot” at least it is is how SmarterStats describes that bots who is hammering my website. So far this month 28Gb of bandwith was used by that bot, just on par with Google. But as it appears in my stats as unknown I can’t stop or at least somehow slow it down.
A couple of days ago I banned the IP address 108.178.59.74 but today I am having an “Unknown bot” from IP 173.199.117.243.
I contacted SingleHop, the organization behing 108.178.59.74 but I didn’t have an answer about the actual people behind this ip. I am now contacting Choopa.com for the other IP.
There are lots of nasty spam bots out there, so my guess is it’s one of them. You could perhaps try to tell it to go away in your robots file, but I doubt it would take any notice.
Do you know what CMS you are using? Assuming that your site is based on ASP.NET, most CMS’s will have caching enabled by default, but it is a good idea to check. Caching will go a long way.
Banning the IP address is probably the best thing you can do. The bot might well come back from another address, so you will need to be vigilant, and keep on banning it. (Normally, I would suggest that you don’t bother, but if it really is stealing 28 GB a month - which sounds like a heck of a lot - you do need to do something about it.)
It is not even a huge website, it’s pretty small with low traffic according to Analytics. I just a look at SmarterStats and found out that traffic from China is causing me those headache. 91% of the traffic is coming from there burning 32Gb of bandwith whereas my website is a directory of business for a region in Chile ! (using 2gb of bandwith) !!
I thought of suggesting the same thing, but I didn’t, mainly because China is a huge market, and there might well be potential visitors from there that you wouldn’t want to lose. Of course, if you’re sure that your site contains nothing that would interest a genuine Chinese visitor, that’s another matter, but I wouldn’t jump to that conclusion.
By the way, the two IP addresses that Corobori mentioned are both in the United States.
Mike,
China is surely a huge market but I doubt Chinese people have much interest in a website in Spanish which is focused on a particular Chilean region. Surely not to attract 977,322 visitors in August so far whereas visitors from Chile were 27,209 (according to SmarterStats). By the way Google Analytics doesn’t “see” any significant traffic from China, actually it says 3 (three) visitors from China for the same period.
I know the 2 IPs I pointed to originally are from the US but I guess I bumped I wouldn’t have noticed them if I hadn’t had the main issue with China which is siphonning off all this bandwidth, I just saw that for august so far it’s 80Gb.
Yes, the only way to stop this completely is to ban everyone … which no doubt will happen some time in the future when we all get sick of the www.