SitePoint Sponsor

User Tag List

Results 1 to 10 of 10
  1. #1
    SitePoint Wizard johntabita's Avatar
    Join Date
    Mar 2001
    Location
    In a big, big house, with lotsa lotsa room
    Posts
    1,062
    Mentioned
    7 Post(s)
    Tagged
    0 Thread(s)

    Will Google spider a site whose domain isn't pointed?

    Not sure if this is the right forum. Will Google spider a site whose domain isn't pointed? If the only way to access it is through the IP address, does the Googlebot still crawl it, or does it skip it until it's pointed?

    Thanks!

  2. #2
    SitePoint Member
    Join Date
    Apr 2013
    Posts
    14
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Yes, Google does crawl via IPs, stopping your http server from responding to IP requests may help.

  3. #3
    Life is not a malfunction gold trophysilver trophybronze trophy
    TechnoBear's Avatar
    Join Date
    Jun 2011
    Location
    Argyll, Scotland
    Posts
    6,207
    Mentioned
    264 Post(s)
    Tagged
    5 Thread(s)
    If you don't want it indexed, the best way to do that is via the robots.txt file.

  4. #4
    It's all Geek to me silver trophybronze trophy
    ralph.m's Avatar
    Join Date
    Mar 2009
    Location
    Melbourne, AU
    Posts
    24,198
    Mentioned
    456 Post(s)
    Tagged
    8 Thread(s)
    Google will happily crawl a site at an IP address only, but only if it can find it. If there are no links to it, Google won't see it.

  5. #5
    SitePoint Wizard bronze trophy PicnicTutorials's Avatar
    Join Date
    Dec 2007
    Location
    Carlsbad, California, United States
    Posts
    3,656
    Mentioned
    15 Post(s)
    Tagged
    0 Thread(s)
    Google can only follow links the same way you can. She cannot look in folders and say oh there it is. If there is no link on the web pointing to it google will not slider it

  6. #6
    SitePoint Wizard johntabita's Avatar
    Join Date
    Mar 2001
    Location
    In a big, big house, with lotsa lotsa room
    Posts
    1,062
    Mentioned
    7 Post(s)
    Tagged
    0 Thread(s)
    Thanks for the replies. Here's what I'm trying to accomplish:

    I don't want the site to get indexed until the customer approves it. We've had situations where a customer doesn't get back to us for over a month, then is upset because we used the wrong verbiage in the title tag and description. Meanwhile, Google has indexed the site and the "wrong verbiage" appears in the SERP.

    If stopping the server from responding to IP requests will prevent indexing, yet still allow me and the client to view the site via IP, then once the client approves the site, we can make it "live" by pointing the domain.

    Will that work? Or is there an easier way that doesn't involve a separate development server and migrating files?

  7. #7
    SitePoint Zealot WebEminence's Avatar
    Join Date
    Jan 2012
    Location
    Chicago, IL
    Posts
    130
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    I've had the same problem, John. Google will index pages on the ip address. I usually don't work on sites for longer than 2 weeks so it's never really a big issue for me. In your case, I would just block search engine with robots.txt. You can use a plugin for this.

    I wonder how Google finds these sites when there are no links. Does it have something to do with Wordpress pinging maybe?

  8. #8
    Life is not a malfunction gold trophysilver trophybronze trophy
    TechnoBear's Avatar
    Join Date
    Jun 2011
    Location
    Argyll, Scotland
    Posts
    6,207
    Mentioned
    264 Post(s)
    Tagged
    5 Thread(s)
    If you're using WordPress, then there should be an option somewhere to disallow indexing. (Sorry - I don't use WP and can't be more specific.) Otherwise, a robots.txt file with
    Code:
    User-agent: *
    Disallow: /
    in the root directory should be enough to stop any reputable bot indexing the site.

    I read somewhere that Google likes to be mentioned by name, but I can't remember where I read that* and don't know how true it is. However, I always add the extra line, just in case.
    Code:
    User-agent: Googlebot
    Disallow: /
    User-agent: *
    Disallow: /
    I've never had any problems with that.

    *except that it was a source I considered reliable at the time.

  9. #9
    SitePoint Zealot WebEminence's Avatar
    Join Date
    Jan 2012
    Location
    Chicago, IL
    Posts
    130
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Yeah, that's true. I forgot. You can turn off search engine indexing under Settings: Reading in Wordpress

    That's probably the easiest option. Set that option and then turn it off when the site goes live. Just make sure not to forget to turn it off later.

  10. #10
    SitePoint Wizard bronze trophy PicnicTutorials's Avatar
    Join Date
    Dec 2007
    Location
    Carlsbad, California, United States
    Posts
    3,656
    Mentioned
    15 Post(s)
    Tagged
    0 Thread(s)
    If its a Wordpress then ya prob settings for anything. But on a normal site if you don't believe google can't spider if no links point to it. Then the easiest way is to just password protect the folder. This is what google suggests by the way. Google cannot crawl a password protected folder. Doing it in cpanel only takes 2 or 3 clicks.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •