SitePoint Sponsor

User Tag List

Results 1 to 3 of 3

Thread: How can I create a robot.txt file using google?

  1. #1
    SitePoint Member mrbetabm's Avatar
    Join Date
    Jul 2012
    Location
    Dhaka
    Posts
    3
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    How can I create a robot.txt file using google?

    I have a new website. I used wordpress platform. Now I want to use robot.txt file for avoid some page google crawling. Is it possible? Can anyone help me?

  2. #2
    Grüße aus'm Pott
    SitePoint Award Recipient Pullo's Avatar
    Join Date
    Jun 2007
    Location
    Germany
    Posts
    2,454
    Mentioned
    39 Post(s)
    Tagged
    1 Thread(s)
    Hi there,

    It is indeed possible.
    Go to your WP dashboard -> Settings -> Privacy, there you have the option "Ask search engines not to index this site".
    As the name suggests, this will ask Google and co. not to index your (entire) site.

    If you want to block search machines only from indexing certain pages, you will have to create a "robots.txt" file and edit it by hand.
    This isn't very difficult however, so if you need a hand, just let us know.
    How well do you know your JavaScript from your jQuery?
    Check out SitePoint's latest JavaScript challenge


    My blog

  3. #3
    SitePoint Member hanisharun.com's Avatar
    Join Date
    Dec 2012
    Location
    Malaysia
    Posts
    12
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    by default WP has a virtual robot file

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •