Improving on a robots text file?

I thought it just needed to be blank? How could you make it better from an SEO point of view please?

If I can guess at what another member might have been thinking five years ago (), I think the idea is that you use robots.txt to exclude any sections of your site you don’t want crawled and indexed (ToS, privacy statement, whatever) so that when the spiders visit, they don’t waste time on these pages, but instead concentrate on those parts of the site you do want indexed.

1 Like

This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.