SitePoint Sponsor

User Tag List

Results 1 to 4 of 4

Hybrid View

  1. #1
    SitePoint Member
    Join Date
    Jan 2000
    Location
    natick, ma, usa
    Posts
    23
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    boy, i'm asking a lot of questions!

    if a site does not have a robots.txt file, and the spider gets a 404 not found error when it comes to try to index the site, does the spider go away, or do they somehow spider it anyway?

    obliocat

  2. #2
    SitePoint Wizard jonese's Avatar
    Join Date
    Jul 1999
    Location
    Powder Springs GA (Atlanta)
    Posts
    1,238
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    they will reord the 404 file not found page. go ahead and try it's amazing th enumber of 404 pages that are indexed by the search engines.

    ------------------
    Eric Jones - WR Moderator / Administrator
    http://www.computermoles.com
    "Digging up the best in computer software!"

  3. #3

    Join Date
    Oct 1999
    Location
    UK
    Posts
    120
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    A robots text file is not neccessary to get spiders to index your site. I promote lots of sites and have never used it with no problems.

    ------------------
    Simon Conroy, Acorn Internet Search Engine Placement
    http://www.topwebsite.co.uk/

  4. #4
    SitePoint Member
    Join Date
    Jan 2000
    Location
    natick, ma, usa
    Posts
    23
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    so, they record the 404 and then index the site anyway? is that right? i added a blank robots file, and looked at my log of people that hit it, which they did, but it's not telling me where they went next. it's like they hit the robots.txt and stop. is that true, or does my tracker just not know how to follow a spider's path through the site?


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •