When I look through my http error logs, there’s always lots of entries for requests to directories or missing pages (usually missing an ID reference eg /products/prodid, where it’s usually /products/prodid1234), resulting in either a “Directory index forbidden by Options directive” error, or just a “file does not exist”. Is it normal to have quite a lot?
I’ve check the on-page html, and request logs in firebug etc many times and can not see my browser making these requests.
I’ve seen that bots will often make up urls that aren’t on-page, just to see what happens, but a lot of these visitors appear to be legit, based on their ips and their browsing behaviour.
How is this happening and is it normal?