so, they record the 404 and then index the site anyway? is that right? i added a blank robots file, and looked at my log of people that hit it, which they did, but it's not telling me where they went next. it's like they hit the robots.txt and stop. is that true, or does my tracker just not know how to follow a spider's path through the site?