Can someone advice me what to do with following please? When I use Google’s tool to remove a url - a 404 Not Found - I put this url in my robot.txt file too to prevent Google accessing it.
So, it that the right way to do it? Or should I not put a block on the url in my robot.txt file and let Google discover the 404 every time? I have noticed many indexed links reported on Google’s tool have all disappeared and I don’t know why.
SEO blogs I read say Google needs new content but it can’t simply be just that. Any advice would be welcome, thanks.
There will be an option in webmaster tool to add your broken links. Just add there… Google will take atleast 1 month before deleting that page from their search engine.
There is no point in blocking Google from accessing ‘dead’ pages in robots.txt - if your server is correctly giving a 404 error, Google will quickly decide that the page is dead and drop it from the index.
Of course, in an ideal world, any pages that are taken off should be 301’d to point to a relevant alternative page, so the only time Google (or anyone else) should get a 404 is by incorrectly making up a URL or following an incorrectly-written link.
- Don’t block Google from accessing your 404s
- If you want search engines and people to go somewhere else when they are looking for that URL, then redirect (using a 301 redirect) to another appropriate live page
A 404 is the correct method of a page not being found (a 410 is the better way when that page will NEVER be found).
Thank you both for your help I will remove those disallowed links from robots.txt for those links that are 404.