Somehow, I ****ed-up and managed to get an outdated part of a client site re-spidered by Google, even though I didn’t want to.
So, two questions…
1 How’d it happen?
2 What’s the best way to fix the problem?
The pre-makeover site was a Wordpress install, publishing into the root.
I re-configged WP to publish to a non-root directory (/old/)and, as far as I am aware, no part of the current site has any links to the old stuff.
But, purely by chance, I today found content in the /old/ directory indexed by Google.
So I’m wondering just how Google got that content? …because, I don’t want that happening again.
And yes, of course I can use a robots file to exclude Google - but usually avoid them because of the problems of directly drawing attention to sensitive content.
So I’m wondering just how Google got that content? …because, I don’t want that happening again.
We can suppose that there’s some link to that section of the website that you don’t know about. Perhaps Wordpress added it without you knowing about it. Another possibility, though I don’t know if it’s real, is that you or someone else accessing that website section was using a pagerank checker (maybe Google’s toolbar). That would result in Pagerank queries to Google, which, in theory, Google could use to learn about new content on the web.
So I’m wondering just how Google got that content? …because, I don’t want that happening again.
Password protection/access for registered members only never fails in that regard.
1 .Create a .htaccess file in /old and put this in it:
deny from all
Or you can set it up so that wordpress sticks a noindex meta tag in every page. Login to wordpress admin panel > settings > privacy > “I would like to block search engines, but allow normal visitors”.
