Google’s URL removal page contains a little bit of handy information that’s not found on their webmaster info pages where it should be.
Google supports the use of “wildcards” in robots.txt files. This isn’t part of the original 1994 robots.txt protocol, and as far as I know, is not supported by other search engines. To make it work, you need to add a separate section for Googlebot in your robots.txt file. An example:
User-agent: Googlebot Disallow: /*sort=
This would stop Googlebot from reading any URL that included the string “sort=” no matter where that string occurs in the URL.
So if you have a shopping cart, and use a variable called “sort” in some URLs, you can stop Googlebot from reading the sorted (but basically duplicate) content that your site produces for users.
Every search engine should support this. It would make real life a lot easier for folks with dynamic sites, and artificial life a lot easier for spiders.
Related posts:
- On-page SEO – The Google Way Mihaela recommends Google’s Search Engine Optimization Starter Guide for all...
- How Google Really Wants You to Optimize Your Site Google is "delighted" when sites are optimized for search --...
- Quick and Easy Graphing with the Google Chart API The Google Chart API makes creating simple charts and graphs...
- Microsoft Kumo Search to Take on Google. Again. Can Microsoft successfully take on Google with its new Kumo...
- Three Hidden Photoshop Pen Tool Tips Jennifer shows you three options and shortcuts that can help...
great find! thanx
October 21st, 2005 at 3:55 pm
Thanks Dan
October 22nd, 2005 at 4:52 pm
useful information. thanks!
October 22nd, 2005 at 7:29 pm
Is this a must add for user using sites like OSCommerce? Is the Sort feature a real cause for duplicate content?
October 23rd, 2005 at 7:17 pm
It could be. Picture a category with one product – no matter how you sort it, it’s the same page. Even with a bunch of items, do you really need the search engines to have every possible order?
October 23rd, 2005 at 8:34 pm
I’ve been using wildcards in robots.txt for…well ever. I had no idea it wasn’t part of the original protocol and I certainly had no idea it was only Google that supports it. Thanks for the info.
October 24th, 2005 at 7:22 am
[...] Dan Thies has found a neat hidden protocol that can be used on your robots.txt file: The Wildcard. User-agent: Googlebot Disallow: /*sort= [...]
October 24th, 2005 at 5:15 pm
[...] I have been using wildcard in my robots.txt file for…ever (this is thedefinition of robots.txt, if you need it). Well, this morning, I happened upon Dan Thies’ Google’s Hidden Protocol post at SitePoint and I thought hmm…interesting. [...]
November 7th, 2005 at 8:48 am
If there is any url in my site containing the word ‘calender’ and I don’t want google to index it than I wil just add
User-agent: Googlebot
Disallow: /*calender
to my robots file,
and it does not matter that under which directory the url with word ‘calender’ is coming, it might be my cg-bin directory.
right ?
December 6th, 2005 at 12:33 pm
[...] SEO Book points to Dan Thies’s finding of a useful but non-standard robots.txt feature supported by the Google spider: [...]
July 11th, 2006 at 11:57 pm
That helps a lot. Thanks for the help!
January 26th, 2007 at 5:13 am
I did this once and at page removal request i got an error for the same.
SEO Services
November 14th, 2008 at 9:54 pm