SitePoint Sponsor |
|
User Tag List
Results 1 to 4 of 4
-
Dec 31, 2007, 06:50 #1
- Join Date
- Sep 2007
- Location
- UP, India.
- Posts
- 25
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
To allow robots to view individual pages in a dir included via PHP or not?
I am using an Apache mod called mod_rewrite. What this does is just this:
eg. www.site.com/friendly-seo-link-here
Now I simply have an index.php page on www.site.com and what that does is just include the right page from a directory called /pages by matching it with whatever comes after www.site.com/
So for eg. if user clicks on www.site.com/article-1
It will include www.site.com/pages/article-1.php in the content area while still displaying the URL www.site.com/article-1
File article-1.php just contains the content which needs to be displayed.
So my question is do I allow the robots to crawl the /pages/ directory or not?
-
Dec 31, 2007, 07:27 #2
- Join Date
- May 2006
- Location
- Aurora, Illinois
- Posts
- 15,476
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
No.
Save the Internet - Use Opera | May my mother rest in peace: 1943-2009
Dan Schulz - Design Team Advisor | Follow me on Twitter
SitePoint References: HTML CSS JavaScript | Become A Guru
WordPress SEO Checklist | What WordPress Plugins Do You Use?
Web Standards Curriculum | Image Free Equal Height Columns
-
Dec 31, 2007, 07:33 #3
- Join Date
- Sep 2007
- Location
- UP, India.
- Posts
- 25
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
Thank you for the answer, Dan Schulz.
-
Dec 31, 2007, 07:51 #4
- Join Date
- May 2006
- Location
- Aurora, Illinois
- Posts
- 15,476
- Mentioned
- 0 Post(s)
- Tagged
- 0 Thread(s)
No problem. The general rule of thumb is to basically allow the robots.txt file to let search engines crawl only those pages you want your regular visitors to see (note: I do tend to block my images folder, though that's a personal preference). Especially where duplicate content (or the mere threat of it) is involved.
Oh, and you can just call me Dan.Save the Internet - Use Opera | May my mother rest in peace: 1943-2009
Dan Schulz - Design Team Advisor | Follow me on Twitter
SitePoint References: HTML CSS JavaScript | Become A Guru
WordPress SEO Checklist | What WordPress Plugins Do You Use?
Web Standards Curriculum | Image Free Equal Height Columns
Bookmarks