Server does not allow Robots, Can you still submit?

[webbgirl]

If a server does not allow Robots, can you still submit a web page to the search engines?

[aspen]

When a host usually says that they do not allow robots it means they do not allow you to run robots. I have never heard of a host not allowing robots to visit - and honestly it'd be fairly complicated for them to do. There is no turn robots on/off feature. They'd have to define specifically which clients or ip addresses they wished to deny.

[Automated]

Might even be a little bit more synister, they could be tring to to keep the bandwidth costs down.

Matthew Taylor

[aspen]

I doubt it, unless its a free host, the bandwidth is paid by the client.

[webbgirl]

Is there a way you can tell if search engine spiders are allowed and do all search engines use spiders to list sites?

Originally posted by aspen
When a host usually says that they do not allow robots it means they do not allow you to run robots. I have never heard of a host not allowing robots to visit - and honestly it'd be fairly complicated for them to do. There is no turn robots on/off feature. They'd have to define specifically which clients or ip addresses they wished to deny.

[aspen]

All search engines use a spider. However there are directories, portals, and search engines.

Yahoo = Directory/Portal
MSN = Portal
Lycos = Portal
AOL = Portal
DMOZ = Directory
AltaVista = SE
Inktomi = SE
Google = SE
HotBot = Portal
Fast = SE

Many people would label all of those as search engines and thats not true, only actual engines will spider your site.