SitePoint Sponsor

User Tag List

Results 1 to 6 of 6
  1. #1
    SitePoint Enthusiast
    Join Date
    Apr 2002
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Server does not allow Robots, Can you still submit?

    If a server does not allow Robots, can you still submit a web page to the search engines?

  2. #2
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,939
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    When a host usually says that they do not allow robots it means they do not allow you to run robots. I have never heard of a host not allowing robots to visit - and honestly it'd be fairly complicated for them to do. There is no turn robots on/off feature. They'd have to define specifically which clients or ip addresses they wished to deny.
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums

  3. #3
    You wanna' peice o' me! Automated's Avatar
    Join Date
    Nov 2001
    Location
    UK
    Posts
    338
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Might even be a little bit more synister, they could be tring to to keep the bandwidth costs down.

    Matthew Taylor
    FreelanceForums.com - Do you Freelance ?
    Visit the Freelance Forums and participate in all areas of Freelance Chat

  4. #4
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,939
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I doubt it, unless its a free host, the bandwidth is paid by the client.
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums

  5. #5
    SitePoint Enthusiast
    Join Date
    Apr 2002
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Server does not allow Robots, Can you still submit?

    Is there a way you can tell if search engine spiders are allowed and do all search engines use spiders to list sites?

    Originally posted by aspen
    When a host usually says that they do not allow robots it means they do not allow you to run robots. I have never heard of a host not allowing robots to visit - and honestly it'd be fairly complicated for them to do. There is no turn robots on/off feature. They'd have to define specifically which clients or ip addresses they wished to deny.

  6. #6
    Serial Publisher silver trophy aspen's Avatar
    Join Date
    Aug 1999
    Location
    East Lansing, MI USA
    Posts
    12,939
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    All search engines use a spider. However there are directories, portals, and search engines.

    Yahoo = Directory/Portal
    MSN = Portal
    Lycos = Portal
    AOL = Portal
    DMOZ = Directory
    AltaVista = SE
    Inktomi = SE
    Google = SE
    HotBot = Portal
    Fast = SE

    Many people would label all of those as search engines and thats not true, only actual engines will spider your site.
    Chris Beasley - I publish content and ecommerce sites.
    Featured Article: Free Comprehensive SEO Guide
    My Guide to Building a Successful Website
    My Blog|My Webmaster Forums


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •