Robots.txt

What is the proper code to create a nofollow in robots.txt for a specific page?

Maybe something like this:

User-agent: *
Disallow: /folder/page.html

That only means that faithful bots like Google’s won’t index it, but others might.

right, useful for me, create nofollow page will get out of lose pagerank? is that true?

In what way? Because of duplicate content?

ok sorry, its silly question…

I always find this guide to robots.txt handy for reference.

Maybe, or maybe not. I just wasn’t sure what you were asking. :slight_smile:

Sorry. But u are wrong. I m agree with that Google’s won’t index it but not agree others might be because there is no specification of code for particular bot which means all the bots are blocked. If u want to restrict only Google’s bot then u have to write following robots.txt

User-agent: Googlebot
Disallow: /folder/page.html

You are quite right that your code is the way to exclude only Googlebot, but that wasn’t quite what ralph.m meant. Googlebot and other ethical bots will obey the robots.txt directive, but there are bots that don’t. You can’t rely on the robots.txt file to keep all bots from indexing parts of your site, because some will simply ignore it.

They aren’t “blocked” in any way. This robots file just tells the bots what you’d prefer. But as TechnoBear says, some bots—like spam bots—don’t care a flying fish what you’d prefer. There is no onus on them whatsoever to obey the robots file, and many don’t.

It’s like putting a sign on your front door “Please don’t rob me” and assuming all the criminals will just shrug and go away. :slight_smile:

Its fine for me, I just care about Google. Bing and Yahoo I am sure also follow this others I don’t really care.