I am having issues with Yandex hitting my site in the thousands within 10 minutes. I read I can use this:
meta name=“Yandex” content=“noindex”
Does anybody know if this works?
Also I read about using a robots.txt file in the root but I don’t totally understand that since I haven’t used that before. Does anybody have any text I can copy and paste for “Yandex” that I can use for that file? I understand that it’s sensitive to spacing and the correct wording, so I am a little lost with that.
I’m not sure about a Yandex-specific meta tag; it’s not something I’ve come across.
You can specify user-agents in the robots.txt file, but robots directives are only advisory: bots can ignore them, although reputable bots like Google will respect them. That will apply equally to robots meta tags and a robots.txt file.
An alternative would be to block Yandex using the .htaccess file, if you’re sure you want to prevent it crawling altogether.
Do you know that if the meta tag (at the top of the page) specific to Yandex works, would it stop them from the rest of the page where I have code for logging activity?
I don’t really mind the bot, but what I don’t like is it becomes a log entry in the database which then makes the logs unreadable with actual users. So I was hoping that the meta tag would stop that, but not sure if it will.
You would probably also need to add “nofollow”, to ensure it doesn’t follow any links on that page. You might find more information on this site: http://www.robotstxt.org/meta.html
I wonder why the Yandex bot is hitting your site so hard. I have always found it to be a well-behaved bot. Of course it could be something else just pretending to be Yandex.
You can ban bots using .htaccess as @TechnoBear says. You would need to check the identifier but something like the following should do the job:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Yandex [NC,OR]
RewriteRule .* - [F]
Alternatively, if the problematic bots are using a narrow range of IP addresses, it might be easier to block those. That way, if it is a bad bot spoofing Yandex, you won’t block the real thing. (You should be able to check if the IP addresses actually belong to Yandex.)