Increase Search Traffic by Getting Your Site Recrawled More Often

Search engine rankings are vital to the success of your website.

And precisely because they’re so important, it’s frustrating when you work hard to improve and optimize your site, only to find that your changes don’t show up in Google for days or even weeks.

This lag is due to the way Google indexes sites on the Internet. It deploys legions of “spiders” that use a sophisticated algorithm that runs across the entire Internet, examining sites and indexing pages based on keywords.

However, there is no fixed period as to when Google’s spiders, or “bots,” crawl your site. In fact, it can vary considerably from site to site.

But you’re not helpless. By taking a few basic steps, you can encourage Google to crawl your site more often–improving your rankings and getting your new pages to show up more quickly.

Reviewing your site’s crawl stats

Since Google’s crawl frequency varies from site to site, to get an idea of how often your site is crawled you first need to look in your Webmaster Tools.

When you first log in to Webmaster Tools, it will guide you through the setup process for your site.

Once you have your site up and running in Webmaster Tools, click Crawl Stats in the left hand menu. Here’s what the stats look like for a bed retail site that I run:

Google only displays crawl data for the last 90 days. On the right, Google tells me that it crawled my bed site an average of 75 times a day, with a high of 322 crawls on the August 8:

As you can see the crawl frequency can vary widely from day to day. This particular site was crawled less than 100 times a day between two and three months ago, with 0 crawls during a couple of days in June and July. Since then it has been crawled much more frequently with an upwards trend over the last month.

There are many reasons for this variance. Site size is one factor. This site is large–it lists thousands of beds and features a product inventory that gets updated regularly. Google tends to recrawl big, frequently updated sites multiple times per day.

The low frequency in July and June represents periods when Google applied a manual action to my site. Since August, those actions have been removed by Google. I’ve also redoubled my marketing efforts, resulting in more inbound links pointing to my site–and more crawls from Google.

That increased crawl rate has been critical in getting my site back up in the rankings after June’s manual actions. During this period I implemented site-wide canonical tags to eliminate duplicate content in an effort to get more of my products organically listed in Google’s search. With over a thousand listed products, getting hundreds of crawls a day helped my SEO efforts pay off.

Getting Google to crawl more often

This is often what most people want to know when they ask how often the big G crawls their site.

The fact is, while some sites get crawled multiple times a day, others won’t get crawled for a month at a time. This can be incredibly frustrating as you continue to add to your site but don’t see the changes reflected in Google’s search results for weeks.

There are multiple ways to encourage Google to crawl your site more frequently.

1. Improve site load times and reduce server errors

Google’s algorithms are more likely to crawl your site if it loads quickly and doesn’t serve up a lot of errors and bad pages.

If your site has connectivity issues, Google will be less likely to crawl your site often. This is why having a good host with reliable servers is crucial.

Furthermore, page load times also have a bearing on how often Google crawls your site. There are multiple ways to optimise page load times which are beyond the scope of this article. However, there is an easy way to see the crawl errors for yourself and see where the problem is.

The Crawl Errors report is located in the left menu under the Crawl section of your Webmaster Tools. As you can see my site looks pretty healthy as far as connectivity is concerned (indicated under the Site Errors heading). From the URL Errors graph, you can see there were two server errors in May and one error in June, and the site is now returning zero errors. This also corresponds with the increased crawl frequency in August as my server errors have been reduced to zero.

2. Update your site frequently

Another way to increase how often Google crawls your site is by updating it frequently.

Google loves regularly updated sites and will reward your efforts by coming back often, making sure the site is fully indexed in its search.

This is straightforward for news sites, but can be tricky for retail sites, such as mine in the above example. Products rarely change–certainly not as often as the news–and it’s difficult for businesses to get around this.

That’s why having a blog can be so important. Blog content can be updated as often as you want. Regularly adding new material will make Google come back and recrawl your site.

3. Create more inbound links

This is one way of increasing your crawl rate that can get you into a lot of trouble if done wrong.

Google frowns on what it considers “inorganic” inbound links to your site, which it deems as spam.

If you try to game Google by buying links, or if you have too many low quality links pointing to your site (such as links from adult or gambling websites), you may find that Google’s spiders run from your site and never return.

However, it’s undeniable that the more authority links you have from other websites pointing to your site, the more likely Google will recrawl it. With newer and fresher links, Google will increase its crawl frequency.

4. Publish a sitemap

There’s a debate as to how effective this strategy is in terms of how often your site is crawled.

Google considers a whole range of factors when determining when to re-crawl your site, and the sitemap is a minor factor in the equation. Site update frequency, inbound links and page load times are much more important.

But by submitting a sitemap to Google, you give its spiders the full layout of your site, making it much easier for it to be crawled.

What’s more, some site map generators allow you to enter a field for how frequently you plan on changing or adding to the content on your site.

Although this alone won’t influence the recrawl rate, if you actually update the site as often as you say you intend to, that will have an impact on how often it’s crawled.

5. Fetch as Google

A final way to get Google to recrawl your site, is to directly request a crawl by submitting a URL in your Webmaster Tools. Under the Crawl menu on the left you’ll see a link for Fetch as Google:

This option allows you to force Google to recrawl your site by entering the full URL to any page in the space provided. This yields almost instant results and is one of the most surefire ways of letting Google know you’ve updated your site.

Google limits you to 10 index submissions per month, per account. If you host a number of client sites in the same Webmaster Tools account, make sure to use your submissions sparingly.

Start by understanding your Crawl Stats

There are clearly many factors that influence how often Google crawls your site.

Trying to understand how these factors influence your rankings starts with getting a clear picture using the Crawl Stats in your Webmaster Tools.

These reports will give you an idea of how you can improve your crawl rate, whether that’s upgrading your servers, submitting a sitemap or increasing your site’s number of inbound links.

Frequently Asked Questions (FAQs) about Increasing Search Traffic and Site Recrawling