boy, i'm asking a lot of questions!
if a site does not have a robots.txt file, and the spider gets a 404 not found error when it comes to try to index the site, does the spider go away, or do they somehow spider it anyway?
they will reord the 404 file not found page. go ahead and try it's amazing th enumber of 404 pages that are indexed by the search engines.
Eric Jones - WR Moderator / Administrator
"Digging up the best in computer software!"
A robots text file is not neccessary to get spiders to index your site. I promote lots of sites and have never used it with no problems.
Simon Conroy, Acorn Internet Search Engine Placement
so, they record the 404 and then index the site anyway? is that right? i added a blank robots file, and looked at my log of people that hit it, which they did, but it's not telling me where they went next. it's like they hit the robots.txt and stop. is that true, or does my tracker just not know how to follow a spider's path through the site?