what if i *don't* have a robots.txt file?

**obliocat** · Feb 2, 2000 16:48

boy, i'm asking a lot of questions!

if a site does not have a robots.txt file, and the spider gets a 404 not found error when it comes to try to index the site, does the spider go away, or do they somehow spider it anyway?

obliocat

**jonese** · Feb 2, 2000 17:06

they will reord the 404 file not found page. go ahead and try it's amazing th enumber of 404 pages that are indexed by the search engines.

------------------
Eric Jones - WR Moderator / Administrator
http://www.computermoles.com
"Digging up the best in computer software!"

**simon c** · Feb 3, 2000 11:34

A robots text file is not neccessary to get spiders to index your site. I promote lots of sites and have never used it with no problems.

------------------
Simon Conroy, Acorn Internet Search Engine Placement
http://www.topwebsite.co.uk/

**obliocat** · Feb 4, 2000 08:19

so, they record the 404 and then index the site anyway? is that right? i added a blank robots file, and looked at my log of people that hit it, which they did, but it's not telling me where they went next. it's like they hit the robots.txt and stop. is that true, or does my tracker just not know how to follow a spider's path through the site?