Do I need an expert for putting Robots.txt in my website?

Recently someone told me that incorporating robots.txt requires an expert to do so. I am confused as I researched this on internet and got some mixed answers at the following links:

1 Like

I don’t have time to read all those articles, but the robots.txt file is basically very simple.

Unless you have unusually complex requirements, you should have no problem creating the file.

1 Like

Thanks for your reply. I will try looking into doing this and hope to have this sorted

1 Like

I agree with the point discussed by TechnoBear. There is no simplest way other than

Yes, I will go with the same. :blush:


No need to hire a external developer to add the robot.txt file into the website. But if you want to add the X robots file to stop bot or crawler to crawl your confidential documents such as pdf, word docs in the website, you might be need the technical stuff from the developer.

1 Like

Yes! sometimes it requires an expert for putting robots,txt in your website

Welcome to the forums, @shcoco58.

Please can you elaborate on your response? Everybody else seems to be agreed that it is quite simple to follow the protocols. Your view is different, so please explain your thinking. Otherwise, your reply is of no real help to anyone.

If you are using wordpress CMS or any other platform to build your website, you can easily create a simple .txt file in notepad and then add some code there and upload it inside the root folder of your domain in cPanel.

In my case, I have added the below code inside my robots.txt file:-

User-agent: *
Disallow: /wp-content/plugins
Disallow: /wp-content/themes