SitePoint Sponsor

User Tag List

Results 1 to 4 of 4
  1. #1
    SitePoint Enthusiast
    Join Date
    Nov 2008
    Location
    Margate South Africa
    Posts
    27
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Robots.txt Horror

    Hi from South Africa,
    I have noticed that on Google Webmaster Tools it says that my blog has robots.txt which means that 15 of my pages do not get visited by the crawlers. What are these? How do you get rid of them? I use Blogger. I am not allowed to give a link yet.
    Thanks
    alanbutt

  2. #2
    Programming Team silver trophybronze trophy
    Mittineague's Avatar
    Join Date
    Jul 2005
    Location
    West Springfield, Massachusetts
    Posts
    17,290
    Mentioned
    198 Post(s)
    Tagged
    3 Thread(s)
    robots.txt files are used to tell Google and other "honest" bots which folders/files the site wants the bots to access and which it doesn't.
    Which are the robots.txt file not allowing? They could be files that you really don't want them to go to. And/or they could be "duplicate" content, i.e. the same post under date/archive/category URLs.

  3. #3
    SitePoint Guru cyjetsu's Avatar
    Join Date
    May 2008
    Posts
    814
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Building a site map for search engines could help.

  4. #4
    In memoriam gold trophysilver trophybronze trophy Dan Schulz's Avatar
    Join Date
    May 2006
    Location
    Aurora, Illinois
    Posts
    15,476
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    cyjetsu, did you even read what he said? He's asking for help about the robots.txt file, not optimizing his site for the search engines.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •