if i post a blog on my website and the blog i disallow from robots.txt of my website, but, do not Noindex it from meta tags and link this blog to a high authority website for fast crawling of robots(googlebot) and suppose the crawler first crawled on that blog and then robots.txt files. Now, what will happen? which instruction googlebot follow first, robots.txt or blog(which has no instruction passes in meta tags) ?
Does this answer your question?
I have to say, though, that this makes no sense to me:
[quote=“Jason_Clark, post:1, topic:212297”]
i disallow from robots.txt of my website, but, do not Noindex it from meta tags and link this blog to a high authority website for fast crawling of robots(googlebot)
[/quote]Why would you disallow a page/section, and then link to it “for fast crawling”?
That is confusing. It took me a few reads to even get what it might be meaning.
I think the keywords here are “hiding” and “URL”.
That is, say I want none of my pages in the /private folder to be known to Google (aka “cloaked”).
So I add that folder to my robots.txt file
Google comes to my site and won’t crawl or index the content blocked by robots.txt
But on another site, someone posts a link to /private/stolen-content.html
Google will not index the stolen-content.html page - BUT - Google will index the page that has the link to that file so the page can still be discovered and is not “hidden”
This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.