Do I need an expert for putting Robots.txt in my website?

Recently someone told me that incorporating robots.txt requires an expert to do so. I am confused as I researched this on internet and got some mixed answers at the following links:

1 Like

I don’t have time to read all those articles, but the robots.txt file is basically very simple.

https://www.robotstxt.org/robotstxt.html

Unless you have unusually complex requirements, you should have no problem creating the file.

1 Like

Thanks for your reply. I will try looking into doing this and hope to have this sorted

1 Like

Hi,
I agree with the point discussed by TechnoBear. There is no simplest way other than https://www.robotstxt.org/robotstxt.html

Yes, I will go with the same. :blush:

2 Likes

No need to hire a external developer to add the robot.txt file into the website. But if you want to add the X robots file to stop bot or crawler to crawl your confidential documents such as pdf, word docs in the website, you might be need the technical stuff from the developer.

1 Like

Yes! sometimes it requires an expert for putting robots,txt in your website

Welcome to the forums, @shcoco58.

Please can you elaborate on your response? Everybody else seems to be agreed that it is quite simple to follow the protocols. Your view is different, so please explain your thinking. Otherwise, your reply is of no real help to anyone.