Alright I guess I'm a bit confused. I was looking around at samples of meta tags to get an idea on what people do. While looking over the homepage coding here at sitepoint I saw
<meta name="robots" content="all" />
So I kind of looked it up and I understand it stops a bot from indexing a page and following links on a page, but what I don't get is does a bot not index a page or follow links without this tag? Just a bit confused why people use this tag.
This tag is useless as it is posted here. This tag should only be used to exclude robots from crawling a page and/or following links on a page. Some webmasters try to use it to tell search engines to crawl a page which is redundant as search engines will crawl every page and follow every link by default. So, to summarize, unless you want to prevent a search engine from indexing a page or following links on a page this tag has no value.
Posting Permissions
Bookmarks