Don't index but crawl for relevancy in adsence

Marketing
1

Can you disallow Google from indexing a folder but allow it to crawl for adsence relevancy in robot.txt?

2

From what I have researched robot.txt will only allow or disallow. So to accomplish what I want to do, it has to be done at the page level with this meta tag

<meta name="ROBOTS" content="NOINDEX,NOFOLLOW">

If I am wrong, let us all know here!

Thanks :slight_smile:

3

Thanks for the info. :slight_smile: Guess I asked the wrong question when googling. I don’t see how it is easier to manipulate all of those robots in robot.txt when one line of meta code on the landing page I am working on will do the trick.

4

Don’t take it badly I was simply joking, I know you know google :slight_smile:
Just because in robots you have everything in one place, so you can make change and target whole directories. If using meta tags it can become a mess to maintain or add new robots.

5

you don’t use search engines? it’s very useful you know. you find such pages as http://www.1your.com/drupal/popularRobotsNamesForRobotsTxtFile

6

And those robots would be named???

7

yes, because the robot for google search and for google adsense are two different ones. That should happen in robots.txt that’s the easiest.