To allow robots to view individual pages in a dir included via PHP or not?

[S.Sameer]

I am using an Apache mod called mod_rewrite. What this does is just this:

eg. www.site.com/friendly-seo-link-here

Now I simply have an index.php page on www.site.com and what that does is just include the right page from a directory called /pages by matching it with whatever comes after www.site.com/

So for eg. if user clicks on www.site.com/article-1

It will include www.site.com/pages/article-1.php in the content area while still displaying the URL www.site.com/article-1

File article-1.php just contains the content which needs to be displayed.

So my question is do I allow the robots to crawl the /pages/ directory or not?

[Dan Schulz]

No.

[S.Sameer]

Thank you for the answer, Dan Schulz.

[Dan Schulz]

No problem. The general rule of thumb is to basically allow the robots.txt file to let search engines crawl only those pages you want your regular visitors to see (note: I do tend to block my images folder, though that's a personal preference). Especially where duplicate content (or the mere threat of it) is involved.

Oh, and you can just call me Dan.