When I logged in to Google Webmaster Tools this morning, I saw the following warning
Some important page is blocked by robots.txt. More Details
Here is my robots.txt for my Magento Based Ecommerce Site - http://www.pudu.com.au/robots.txt
Could you please advice if this is a Good Robots.txt from an SEO Perspective.
Thanks,
Nuk
Without knowing what’s in each of those folders, it’s difficult to know whether it’s good or not. (This is why I hate CMSs, because you never have a damn clue what they’re doing…)
If there is any content in any of those folders that you want search engines to be able to access, remove those folders from the list.
To be honest, I really don’t see any need for any of it. What’s the point? Google isn’t going to waste its time reading Javascript or CSS files, you don’t need to tell it not to. It isn’t going to read any pages that aren’t publicly accessible via clicking on links, so you don’t need to ‘block’ folders that don’t have publicly accessible pages because it will never find them. If you’re doing it from a security point of view then you’re just shooting yourself in the foot, because ‘bad’ robots won’t pay any attention to the file, except for possibly to find out where you don’t want them to go so that they can head straight there.
The only case where I’ve found robots.txt to actually have a positive use is where I’ve had PDF downloads available, and wanted search engines to direct people to the HTML versions of the pages instead.
User-agent: *
Disallow: /index.php/
Disallow: /.js$
Disallow: /.css$
Disallow: /checkout/
Disallow: /tag/
Disallow: /catalogsearch/
Disallow: /review/
Disallow: /app/
Disallow: /downloader/
Disallow: /js/
Disallow: /lib/
Disallow: /*.php$
Disallow: /pkginfo/
Disallow: /report/
Disallow: /skin/
Disallow: /var/
Disallow: /catalog/
Disallow: /customer/
this is not standard robots text file but i use it for mysite.I have no idea where i have taken this.hope this helps.
If you don’t know what the code does, please don’t suggest that other people use it. It could have significant consequences and severely damage their site’s performance if you can’t explain what it does or why it would help.
where is my last post.??i was just about to edit !
I’ve removed it. It was a bit too… err… too… direct 
Unfortunately I don’t know Magento that well to know if your txt file is the best or not. I can tell you that, in principle, it looks correct although there are a couple of lines from bharatmax that could be happily added to your file
Disallow: /js/
Disallow: /lib/
Disallow: /skin/
Basically, a txt file allows any agent (that include robots) to read all the files and folders unless your forbid it (that is you disallow it). Well, at least that’s the way it should be. Some bots ignore the txt file and visit whatever they want to visit.
From the SEO perspective, Javascript files, CSS files or the checkout provide no value. In the case of the checkout it is even good because it is more than unnecessary. The process will take the customer to a protected are to include his preferred method of payment.
Folders skin (which only include files related to the look of your site, but not information about your producs), lib (where your programming libraries would go) or JS (where many, if not all, of your javascript files are kept) can be happily excluded from the bot’s curiosity.
Other folders could be up to discussion and I’m a bit uncertain about them, such as catalogsearch or tags. It depends on what it goes there, they can be excluded.
Providing a xml map with the structure of your site is a must!
In wordpress tags are useful.but for magento nobody used for any SEO advantages.I have seen many sites using Magento but none used tags.I dont know why and how it works for magento.
It may good to look at it at some point. Maybe they are underused and that could make a difference in our efforts to reach number one. I guess it will need some investigations 
you should disallow all those folders which shouldn’t be indexed by search engines. for e.g admin folder.