Don't index but crawl for relevancy in adsence

Can you disallow Google from indexing a folder but allow it to crawl for adsence relevancy in robot.txt?

From what I have researched robot.txt will only allow or disallow. So to accomplish what I want to do, it has to be done at the page level with this meta tag

<meta name="ROBOTS" content="NOINDEX,NOFOLLOW">

If I am wrong, let us all know here!

Thanks :slight_smile:

Thanks for the info. :slight_smile: Guess I asked the wrong question when googling. I don’t see how it is easier to manipulate all of those robots in robot.txt when one line of meta code on the landing page I am working on will do the trick.

Don’t take it badly I was simply joking, I know you know google :slight_smile:
Just because in robots you have everything in one place, so you can make change and target whole directories. If using meta tags it can become a mess to maintain or add new robots.

you don’t use search engines? it’s very useful you know. you find such pages as

And those robots would be named???

yes, because the robot for google search and for google adsense are two different ones. That should happen in robots.txt that’s the easiest.