Using robots.txt or robots tag is better as a on-page SEO technique?
Both do pretty much the same thing so won’t make any difference.
Which you use depends upon what your requirements are for directing robots.
Though it’s a good idea to have a robots.txt even if there is not much specific in there, just because so many bots look for it, so it will avoid filling your logs with 404 errors.
1 Like
I’m not sure I would consider either to be an “on-page SEO technique”. Their purpose is simply to advise bots if there are any pages or sections of your site which you do not wish to have crawled or indexed.
Google recommends the meta tag as the more reliable method to ensure pages are not indexed by mistake: https://support.google.com/webmasters/answer/6062608?hl=en
As @SamA74 says, it’s good practice to include a robots.txt file for all those “good” bots which look for one before crawling your site.
1 Like
robots.txt is a one of the important concept in On-Page Optimization. When search engine crawlers come to your site it will be looking for some special file. i.e, call robots.txt file. This file tells to search engine spiders which pages of your site should be index and which pages of your site should be ignored.
This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.