What is the bot up to ??

[jehoshua]

Hi,

Noticed a few "403" messages in the web server logs. The culprit is:

IP 66.147.154.3
canonical name wfp2.almaden.ibm.com.
aliases 3.154.147.66.in-addr.arpa

The actual log entries are:

66.147.154.3 - - [27/Apr/2004:12:48:10 -0400] "GET /robots.txt HTTP/1.0" 403 - "-" "http://www.almaden.ibm.com/cs/crawler [c01]"
66.147.154.3 - - [27/Apr/2004:12:48:39 -0400] "GET / HTTP/1.0" 403 - "-" "http://www.almaden.ibm.com/cs/crawler [c01]"

I can't work out what they were trying to do, there are no (restrictive) file permissions on robots.txt. In fact, the very next entry in the logs was:

65.54.164.112 - - [27/Apr/2004:13:08:27 -0400] "GET /robots.txt HTTP/1.0" 200 54 "-" "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"

.. a nice (friendly) "200" message, and yet the request was exactly the same:

"GET /robots.txt HTTP/1.0" ??

Beats me, I know some crawlers/bots can easily disguise what they are _really_ doing, so what was this one doing, I wonder ??

Peter

[jehoshua]

Hi,

I think I need to ban this IP address, today more "403", the web server logs:

[Wed Apr 28 15:48:11 2004] [error] [client 66.147.154.3] client denied by server configuration: /home/username/public_html/403.shtml
[Wed Apr 28 15:48:11 2004] [error] [client 66.147.154.3] client denied by server configuration: /home/username/public_html/se
[Wed Apr 28 15:48:01 2004] [error] [client 66.147.154.3] client denied by server configuration: /home/username/public_html/403.shtml
[Wed Apr 28 15:48:01 2004] [error] [client 66.147.154.3] client denied by server configuration: /home/username/public_html/_cmdlogin
[Wed Apr 28 15:47:51 2004] [error] [client 66.147.154.3] client denied by server configuration: /home/username/public_html/403.shtml
[Wed Apr 28 15:47:51 2004] [error] [client 66.147.154.3] client denied by server configuration: /home/username/public_html/index.html
[Wed Apr 28 15:47:42 2004] [error] [client 66.147.154.3] client denied by server configuration: /home/username/public_html/403.shtml
[Wed Apr 28 15:47:42 2004] [error] [client 66.147.154.3] client denied by server configuration: /home/username/public_html
[Wed Apr 28 15:47:32 2004] [error] [client 66.147.154.3] client denied by server configuration: /home/username/public_html/403.shtml
[Wed Apr 28 15:47:32 2004] [error] [client 66.147.154.3] client denied by server configuration: /home/username/public_html/robots.txt

What do you think, website abuse ??

Peter

[burkul]

Do you use any .htaccess files on your server? or do you banned anything on your httpd.conf file??

BTW this is not the place to ask this question.

Can some admin move this under a proper topic.

[jehoshua]

Hi,

Originally Posted by burkul

Do you use any .htaccess files on your server? or do you banned anything on your httpd.conf file??

We don't have access to httpd.conf, however we can do (almost) anything with .htaccess.

Originally Posted by burkul

BTW this is not the place to ask this question.

Well, I disagree, this forum had the most hits when I did a search on bots,crawlers,spiders,user agents, so it is therefore the appropriate forum. After all, the forum is SEO, and SE 'crawling' can only ever happen when an "agent" visits your site.

Peter

[burkul]

"client denied by server configuration"
I have a lot of these on my error_logs because I ve banned some agents in .htaccess, i am pretty sure that this has sth to do with your server access configuration which by default there are no restrictions in apache and...

that is why I thought this is not the right place to disscuss server configuration .

whatever

peace.