I have a new website. I used wordpress platform. Now I want to use robot.txt file for avoid some page google crawling. Is it possible? Can anyone help me?
Hi there,
It is indeed possible.
Go to your WP dashboard -> Settings -> Privacy, there you have the option “Ask search engines not to index this site”.
As the name suggests, this will ask Google and co. not to index your (entire) site.
If you want to block search machines only from indexing certain pages, you will have to create a “robots.txt” file and edit it by hand.
This isn’t very difficult however, so if you need a hand, just let us know.
by default WP has a virtual robot file
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/