robots.txt should not be used to ban large portions of your site...

Printable View