SitePoint Sponsor

User Tag List

Results 1 to 8 of 8
  1. #1
    SitePoint Enthusiast
    Join Date
    Nov 2009
    Posts
    90
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Hidden test domains

    I have a (test) domain name I use for development. Perhaps I should be using a sub-domain name. But I don't.

    I know of no published link to this domain anywhere. Sometimes--while developing a new website--I forget to put a meta NOFOLLOW element in the headers of the new pages I'm generating.

    Interestingly, keyword searches on the content of those new and not-yet-published websites often end up indexed on Google. How did Google ever know about that domain in the first place, when I never (ever) published a link to it?

    I suppose it only has to be spidered once and then it's a known entity for ever. But I am still curious to know how they ever found out about my test domain in the first place. Is NOFOLLOW the only way to have a test domain? Or are sub-domains a better way to go?

  2. #2
    SitePoint Addict bronze trophy
    Join Date
    Mar 2013
    Location
    Oneonta, NY
    Posts
    307
    Mentioned
    2 Post(s)
    Tagged
    0 Thread(s)
    Very good question pittendrigh,

    I was not able to find a definitive answer on this but have seen the same activity when I build websites. These sites are not linked to anything (that I know of or that I can find) and yet they appear in Google search results despite the claim that Googlebot finds pages based on other links.

    Here is an interesting article that may point you in the right direction.

    http://www.seroundtable.com/archives/016288.html

    Would be interesting to hear what you find out.

    Best,
    Shawn

  3. #3
    SitePoint Enthusiast
    Join Date
    Nov 2009
    Posts
    90
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Ah. The Google toolbar. I have never used it. But some of my customers probably have. So when I sent my customer an email that said: "look at my test domain and tell me what you think" ......... then their browser makes Google aware of that URL. Once aware once they know about it forever.

  4. #4
    Barefoot on the Moon! silver trophy Force Flow's Avatar
    Join Date
    Jul 2003
    Location
    Northeastern USA
    Posts
    4,617
    Mentioned
    56 Post(s)
    Tagged
    1 Thread(s)
    Sometimes WHOIS records are replicated, so eventually google crawls finds the linked A records and follows them.

    If you don't want them to be indexed or visited, the only sure way to prevent access is to throw up an htaccess username/password prompt. Most hosts offer this through the control panel named something like "protected folder".
    Visit The Blog | Follow On Twitter
    301tool 1.1.5 - URL redirector & shortener (PHP/MySQL)
    Can be hosted on and utilize your own domain

  5. #5
    SitePoint Enthusiast
    Join Date
    Nov 2009
    Posts
    90
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Hadn't thought about passwords. That would be pretty bomb proof.

  6. #6
    Programming Since 1978 silver trophybronze trophy felgall's Avatar
    Join Date
    Sep 2005
    Location
    Sydney, NSW, Australia
    Posts
    16,869
    Mentioned
    25 Post(s)
    Tagged
    1 Thread(s)
    A simple Deny request in the robots.txt file will work with the legitimate search engines without requiring a password.
    Stephen J Chapman

    javascriptexample.net, Book Reviews, follow me on Twitter
    HTML Help, CSS Help, JavaScript Help, PHP/mySQL Help, blog
    <input name="html5" type="text" required pattern="^$">

  7. #7
    SitePoint Enthusiast
    Join Date
    Nov 2009
    Posts
    90
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    This (robots.txt) is the best solution. NOFOLLOW in the header is too hard to control with generated pages that rely on config files or database values......at least in the chaos of development time. Robots.txt tends to be stable throughout all of that.

  8. #8
    SitePoint Member belansus's Avatar
    Join Date
    Aug 2013
    Location
    Newcastle
    Posts
    11
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Not sure if it what your looking for, but I redirect all IPs apart from my own.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •