boy, i'm asking a lot of questions!

if a site does not have a robots.txt file, and the spider gets a 404 not found error when it comes to try to index the site, does the spider go away, or do they somehow spider it anyway?

obliocat