Spiders from Russia

I’ve been getting a fair amount of spiders from Russia hitting my website?

With Russia being one of the Cyber-Criminal capitals of the world, should I be concerned?? :-/

Debbie

no

let them have access to your site just like everybody else

How do you know that these are spiders?

It is true that there is a fair amount of malicious activity coming from Russia. If you see any negative effect like many fake registrations, SPAM or just load on your servers you might want to consider preventing access from Russia all together.
This is assuming your business does not target the Russian audience and you do not care about being indexed by Russian search engines like Yandex.

It’s called the World Wide Web for a reason. Besides, there are enough spammers elsewhere in the world, including America and Britain.

This is an empty statement that does not really help anybody.
I don’t understand your point. Are you saying that because spam is a global problem we should leave it to be?

If a site owner spends 1 hour a day removing fake registrations and blocking a country (which no HUMAN visitor comes from) prevents half of them it frees up 30 minutes a day, 3.5 hours a week… Why not do that?

BTW I don’t know what statistics you are looking at but Britain are not too strong in spam.

In my opinion the reason the Internet has become so powerful is because it brings the world together. Any site or service that chooses to block people because they’re from an area where spam is rampant is not a site that I want to use.

My comment has a message behind it, and that message does nothing but help when discussing the Internet.

Or, you know, you could always beef up your validation rules and work on your own anti-spam measures?

Any Web Developer worth their money should be capable of reducing spam significantly. I work for a company that holds a number of large websites and we are regularly targeted by spammers. The last time I had to clear spam out of our database was last year because one of us accidentally uploaded the wrong form.

From a technical perspective, blocking by country is easy to get around and I’d be shocked if a spammer couldn’t get around it. Hell, I can get around most services like YouTube through nothing more than a freely-available Firefox extension.

I never stated any statistics. I know for a fact that people in Britain spam because I’ve worked with enough companies that employ dodgy tactics, such as mass-emailing, multiple account creation through proxy services, deals with virus manufacturers and link farms, etc.

As you stated, spam is a worldwide problem and you simply won’t get rid of it by blocking certain parts of the world.

First of all, Debbie, sorry for jumping your thread but it is an interesting discussion.

I see now the difference in our perspectives.
You work for a company with enough resources and knowledge to implement solutions. I work for a company that helps those people that don’t have the resources or the knowledge to deal with the problem by themselves.
You are discussing the Internet and I am discussing down to earth particle security measures that will help you run your web site.

Although our solution does not involve geo-blocking, for many people trying to do business on the Internet it works. Why? You say you would not want to use a site that blocks certain geographies. But if you live in Britain, and the web site targets Britain you will not be blocked. Why would people work overtime to deal with spam coming from a country that does in bring them any income? For the sake of the Internet?

Think regular people, your average Joe with an average web site. He simply does not have the tools, knowledge or time to do it by his own. Furthermore, in my previous company we offered web application security solution for Fortune 500 companies and guess what - they also had geo based rules. Not necessarily blindly blocking but specific security measures per location. If you are legit you will never encounter them but if you seem risky you will have to go through a more tight screener. Don’t you think this is much better than having all your users struggling to fill out a CAPTCHA on every form? I assure you that more people will not want to use your site if you have a CAPTCHA everywhere than if you geo block. I think you are an exception here having such strong negative feelings about geo based rules.

I agree spam will not go away by blocking countries but complete solutions cost money that some people do not have. Why should these people care about complete solutions if they can easily get rid of half the problem by blocking bots coming from a country that they do not target. They don’t want spam to go away from the Internet, they want spam to go away from their own site.

Finally, regarding your comment about how easy it is to bypass geo-based rules. This is true if someone specifically targets you and goes the extra mile to understand your rules and bypass them. For the majority of site, spam is generic and does not even notice or care if it is blocked. Most sites are not specifically targeted.

I haven’t been spammed yet. knocks on wood

However, I’ve just noticed that out of the couple of people who visit my client’s site, it seems like in the last week they are all bots from Russia.

So that made me nervous that the website was getting indexed like crazy in Russia and maybe that would lead us to getting hammered by spammers.

(This is a U.S. site, so why would anyone in Russia care that we exist?)

Debbie

Debbie,

Malicious bots will reach your site one way or another even if you are not indexed by Russian search engines. They go through major search engines, IP scans and domain lists to expand their coverage.
If it is a search engine it is natural that they would want to index your site to improve their search results (hey, people for Russia look for US sites as well).
The only thing I would suggest looking into is making sure that these are really indexing bots from search engine companies and not something else disguising as indexing bots.

Things you can do by yourself:

  1. Look up their IPs at DomainTools | Whois Lookup, DNS Lookup, Reverse Whois Lookup and see if the IPs are owned by a real company
  2. Take a look at the logs and see if hits have anything suspicious in them (strange URLs or strange parameters)
  3. Measure the frequency of visits and see if it matches your site profile, visitor count and the rate in which the content changes (how dynamic it is)

I hope this helps.
Let me know if you have further questions.

Eldad

I agree with you in principle, but I must say that I am one of two Web Developers fresh out of university. If anything, we work for a company with very few resources when compared to many of the other development shops around.

We operate a legal website for the UK, but around 20% of our returning organic traffic comes from outside of the UK. Sure, we’re a content site that also offers legal services, but there are a lot of people outside of the area we cover that regularly visit our website. We’ve even landed a few decent deals with clients outside of the UK.

I agree that it’s an easy solution that will work, but it is a poor solution that a large number of people would not agree with if they knew. In my view, there is a reason why Web Developers exist and if you have a spam problem that you cannot fix then you need a Web Developer to come in and help out. If the exhaust on your car broke you wouldn’t attach a plastic tube and go on with your business, you’d go to a mechanic and get the job done properly.