Recently someone told me that incorporating robots.txt requires an expert to do so. I am confused as I researched this on internet and got some mixed answers at the following links:
I don’t have time to read all those articles, but the robots.txt file is basically very simple.
Unless you have unusually complex requirements, you should have no problem creating the file.
Thanks for your reply. I will try looking into doing this and hope to have this sorted
I agree with the point discussed by TechnoBear. There is no simplest way other than https://www.robotstxt.org/robotstxt.html
Yes, I will go with the same.
No need to hire a external developer to add the robot.txt file into the website. But if you want to add the X robots file to stop bot or crawler to crawl your confidential documents such as pdf, word docs in the website, you might be need the technical stuff from the developer.
Yes! sometimes it requires an expert for putting robots,txt in your website
Welcome to the forums, @shcoco58.
Please can you elaborate on your response? Everybody else seems to be agreed that it is quite simple to follow the protocols. Your view is different, so please explain your thinking. Otherwise, your reply is of no real help to anyone.