Yes, that is the standard way. You need to create the robots.txt file.
You can also control this using meta tags, but the robots.txt is the best method.
If it’s absolutely crucial that your site isn’t indexed, you can always add some code to check the User-Agent field that is submitted to the web server, and check to see if it’s a known search engine crawler, and if so, don’t output any HTML.
You can also control this using meta tags, but the robots.txt is the best method.
Like this???
<meta name=”robots” content=”noindex”>
If it’s absolutely crucial that your site isn’t indexed, you can always add some code to check the User-Agent field that is submitted to the web server, and check to see if it’s a known search engine crawler, and if so, don’t output any HTML.
Checking the useragent for any purpose other than collecting statistics isn’t a good idea as any such attempted block can be easily bypassed simply be making a small change to the useragent (and you wouldn’t want to check thousands of search engine useragents in the first place).
Those two lines in the robots.txt file is all that is needed to stop legitimate search engines from indexing your site.
Also if you don’t have any incoming links from an indexed web page then no spiders/robots will find your site anyway. That’s why most search engines have indexed less than 10% of the web and even Google hasn’t found 20% of web pages yet.
You have no control over incoming links to your site whatsoever. You can’t even tell where the visitors to your site come from unless they tell you (via the user controllable REFERRER header).
If you block search engines from listing your site via robots.txt (which is the recognised way) then its highly unlikely anyone will find the site anyway… so how will they link to it? A lucky guess on your URL maybe but thats highly unlikely… akin to search for a needle in a haystack… and the internet is one very large haystack
As long as you have got the robots.txt in place you should be fine.
Blocking or restricting web spiders can prevent your website from Bandwidth lose.
Allowing all web spiders can eat your Bandwidth dramatically.
Select the useful spiders list and allow them to crawl into your website.
robot.txt is an exclusion protocol used to prevent the web spiders to access certain pages of a site. This can be carried out in the following format.
Allows web robots to visit all files
User-agent:*
Disallow:
Allows all web robots out
User-agent:*
Disallow: /
By specifying the directory name between two slashes in Disallow, denies the web spiders to access to those specified directory.
