Results 1 to 2 of 2
Nov 30, 2012, 06:08 #1
- Join Date
- Oct 2012
- 0 Post(s)
- 0 Thread(s)
Problem with robots.txt file coding ?
My question is like very simple but for me a very big cause to think .
In robots.txt file if i indicate one of my URL having like www.example.com/category/internetmarketing.php , so if do use this particular page not to crawl in Google . Then is it possible that only this URL will be considered for not to crawl as it having main category page also.
According to my it will consider main url like www.example.com/category page.
So will please somebody clarify me how this problems will be removed .
Last edited by TechnoBear; Nov 30, 2012 at 06:17. Reason: Example URLs delinkified
Nov 30, 2012, 06:23 #2
- Join Date
- Jun 2011
- Argyll, Scotland
- 265 Post(s)
- 5 Thread(s)
If I've understood you correctly, you want Google to crawl your "category" directory, but not the single page within it called internetmarketing.php.
In that case, all you need in your robots.txt file is:
It will continue to crawl every other page in "category", (provided it can access them, of course).
If you want to block all search engines, and not just Google, then use:
Remember, this will only stop well-behaved bots whih respect the robots.txt file; it won't protet you from bad bots.