Hi All,
Thanks for sharing this information here. Very useful indeed.
Regards,
D Sarathy.
RankQuest
thank Stymiee. It will help me a lot.
very very thanx for information 
Stymiee is a good SEO!, i think he is better than Matt Cutts
This is good stuff. I’m not even a beginner and I learned something. This is what makes Sitepoint great when people like you take the time to help us all while asking for nothing in return! Keep it up.
This post just rocks! I keep getting more and more great info. thanks all!
Insanely useful post, thank you so much.
Basically this helps get rid of duplicate content, low-quality content, css, javascript, php, etc… but does allow search engines to read the articles, find images, find pdfs, etc.
# Allow all
User-agent: *
Disallow:
# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /comments/
Disallow: /js/
Disallow: /css/
Disallow: /about/legal-notice/
Disallow: /about/copyright-policy/
Disallow: /about/terms-and-conditions/
Disallow: /about/feed/
Disallow: /about/trackback/
Disallow: /contact/
Disallow: /stats
Disallow: /tag
Disallow: /category/uncategorized
Disallow: /wp-
# disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
# disallow all files with? in url
Disallow: /*?*
# disallow all files in /wp- directorys
Disallow: /wp-*/
# disallow archiving site
User-agent: ia_archiver
Disallow: /
# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow:
Allow: /*.gif$
Allow: /*.png$
Allow: /*.jpeg$
Allow: /*.jpg$
Allow: /*.ico$
Allow: /*.jpg$
Allow: /images/
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Crawl-delay: 2
phpBB robots.txt
# Allow all
User-agent: *
Disallow:
Disallow: /js/
Disallow: /css/
Disallow: /cgi-bin/
Disallow: /db/
Disallow: /admin/
Disallow: /cache/
Disallow: /includes/
Disallow: /templates/
Disallow: /V
Disallow: /stats
Disallow: /post
Disallow: /member
Disallow: /mx_
Disallow: /index.php?
Disallow: /posting.php
Disallow: /groupcp.php
Disallow: /search.php
Disallow: /login.php
Disallow: /privmsg.php
Disallow: /post
Disallow: /profile.php
Disallow: /memberlist.php
Disallow: /faq.php
Disallow: /archive
# disallow archiving site
User-agent: ia_archiver
Disallow: /
# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow:
Allow: /*.gif$
Allow: /*.png$
Allow: /*.jpeg$
Allow: /*.jpg$
Allow: /*.ico$
Allow: /*.jpg$
Allow: /images/
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Crawl-delay: 2
For SEO Optimized phpBB
# Allow all
User-agent: *
Disallow: /js/
Disallow: /css/
Disallow: /cgi-bin/
Disallow: /db/
Disallow: /admin/
Disallow: /cache/
Disallow: /includes/
Disallow: /templates/
Disallow: /V
Disallow: /stats
Disallow: /post
Disallow: /member
Disallow: /mx_
# disallow these urls
Disallow: /viewtopic.php
Disallow: /viewforum.php
Disallow: /index.php?
Disallow: /posting.php
Disallow: /groupcp.php
Disallow: /search.php
Disallow: /login.php
Disallow: /profile.php
Disallow: /memberlist.php
Disallow: /faq.php
Disallow: /common.php
Disallow: /index.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /privmsg.php
Disallow: /viewonline.php
# disallow urls starting with quote
Disallow: /"
# disallow all files with a ? in url
Disallow: /*?*
# disallow all files ending in specific extension
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$
# disallow archiving site
User-agent: ia_archiver
Disallow: /
# allow google image bot to search all images
User-agent: Googlebot-Image
Disallow:
Allow: /*.gif$
Allow: /*.png$
Allow: /*.jpeg$
Allow: /*.jpg$
Allow: /*.ico$
Allow: /*.jpg$
Allow: /images/
# allow adsense bot on entire site
User-agent: Mediapartners-Google*
Disallow:
User-agent: *
Crawl-delay: 2
Hi, thanks for the tips its very useful.
thanks for the valuable tips.
this answers most of my questions. perfect FAQ 
Nice robots.txt produke!
Google has written some articles recently about using Robots.txt and meta tags to control robots.
- Controlling how search engines access and index your website
- The Robots Exclusion Protocol
- robots.txt analysis tool
- Googlebot
- Inside Google Sitemaps: Using a robots.txt file
- All About Googlebot
Non-Google Resources
[list][]Creating the ultimate WordPress robots.txt file
[]Create a robots.txt file and increase your search engine rankings
[*]Robots.txt info on WordPress.org[/list]
The amount of information is golden… Its an absolute read for all the newbies out there…
I agree totally with Stymiee in regards to Link Popularity
1 of my links I was actively promoting jumped up to a PR2 & when the ad expired, the following week it dropped down to a PR5 or 6
Awesome Forum…Keep it up guys & gals
Never imagine that robot.txt has a lot of thing to do with many things.
Thanks for sharing.
“Build your website for human beings, not search engines!”
I like this quote. Sometimes we lose track of this simple fact. Nice one.
Well done post. Covers most topics that people should know about search engines.
really helpful tips
Great article covers it all.
Gonna read this up and gain some(or more) knowledge on SEO…thank you!
A great share , i really appreciate this share thanks.
This is very true for the people who don’t do their homework and “think they have search engines figured out”.
I see many small web developers in my area sticking every small rural town and a combination of web design or hosting about 40 times on a page.
it’s disgusting to read and look at.