How to Get Your Site Indexed by Google

By Sam Gooch

How to get your site indexed by Google

This article is part of an SEO series from WooRank. Thank you for supporting the partners who make SitePoint possible.

If it’s not already, organic search traffic needs to be a priority for your digital marketing plan. More than half of traffic on the Internet comes from search engines (it could be as high as 60%). Organic search traffic is also hugely important in generating online sales. Of course, you realize your SEO is a priority — but where to start? All SEO starts with getting your website found, crawled and indexed by search engine robots.

In this piece you’ll learn about the technical and on page aspects of SEO and how you can use them to attract the attention of Google, Bing and other search engines.

Step 1: On Page SEO

The first step to getting found by search engines is to create your pages in a way that makes it easy for them. Start off by figuring out who your website is targeting and decide what keywords your audience is using in order to find you. This will determine which keywords you want to rank for. Best practice is to target long-tail keywords as they account for the vast majority of search traffic, have less competition (making it easier to rank highly) and can indicate a searcher is in-market. They also have the added bonus of getting more clicks, having a higher click-through rate (CTR), and more conversions.

There are quite a few free keyword research tools available out on the web.

Once you have your target keywords, use them to build an optimized foundation for your pages. Put your keywords into these on page elements:

Title tag: Title tags are one of the most important on page factors search engines look at when deciding on the relevance of a page. Keywords in the title tags tell search engines what it will find on the page. Keep your title tags 60 characters or less and use your most important keyword at the beginning. A correctly used title tag looks like this:

<title>Page Title</title>

Meta description: Meta descriptions by themselves don’t have much of an impact on the way search engines see your page. What they do influence is the way humans see your search snippet — the title, URL and description displayed in search results. A good meta description will get users to click on your site, increasing its CTR, which does have a major impact on your ranking. Keywords used in the descriptions appear in snippets in bold so, again, use yours here.

Page Content: Obviously, you need to put your keywords in your page content. Don’t stuff your content though, just use your keyword 3-5 times throughout the page. Incorporate some synonyms and latent semantic indexing (LSI) keywords as well.

Add a blog: Aside from the more stereotypical content marketing SEO benefits, blogs are crawling and indexing powerhouses for your site. Sites that have blogs get an average of:

Adding and updating pages or content to your site encourages more frequent crawling by search engines.

Step 2: Technical SEO

Robots.txt

After you’ve optimized your on page SEO factors for your target keywords, take on the technical aspects of getting Google to visit your page. Use a robots.txt file to help the search engine crawlers navigate your site. Very simply, a robots.txt file is a plain text file in the root directory of your website. It contains some code that dictates what user agents have access to what files. It usually looks something like this:

User-agent:*

Disallow:

The first line, as you can probably guess, defines the user agent. In this case the * denotes all bots. Leaving the Disallow line blank gives bots access to the entire site. You can add multiple disallow lines to a single user-agent line, but you must make a separate disallow line for each URL. So if you want to block Googlebot from accessing multiple pages you need to add multiple disallows:

User-agent: Googlebot

Disallow: /tmp/

Disallow: /junk/

Disallow: /private/

Do this for each bot you want to block from those pages. You can also use the robots.txt file to keep bots from trying to crawl certain file types like PowerPoints or PDFs:

User-agent:*

Disallow: *.ppt$

Disallow: *.pdf$

To block all bots from your entire site, add a slash:

User-agent:*

Disallow: /

It’s good practice to block all robots from accessing your entire site while you are building or redesigning it. Restore access to crawlers when your site goes live or it can’t be indexed. Also be sure that you haven’t blocked access to Schema.org markup or it won’t show up in Google’s rich search results.

If you have a Google Search Console account, you can submit and test your file to the robots.txt Tester in the Crawl section.

robots.txt Tester

XML Sitemaps

XML sitemaps are, like robots.txt files, text files stored in your site’s directory. This file contains a list of all your site’s URLs and a little extra information about each URL’s importance, last update, update frequency and if there are other versions of the page in a different language. The sitemap encourages search engines to crawl your pages more efficiently. Sitemaps include the following elements:

  • <urlset> – The opening and closing line of the sitemap. It’s the current protocol standard.
  • <url> – The parent tag for each URL in your site. Close it with </url>
  • <loc> – This is the absolute URL where the page is located. It’s absolutely vital that you use absolute URLs consistently (http vs. https., wwww.example.com or example.com, etc.)
  • <lastmod> – The date on which the page was last updated. Use YYYY-MM-DD format.
  • <changefreq> – The frequency at which you make changes to the file. The more frequently you update a page, the more often it will be crawled. Search engines can tell when you’re lying, though, so if you don’t change it as frequently as you set here, they will ignore it.
  • <priority> – A page’s importance within the site, ranging from 0.1 to 1.0.
  • <xhtml:link> – This provides information about alternate versions of the page.

When properly implemented, your sitemap should look like this:

XML sitemap example

If you have a large or complicated site, or don’t want to make a sitemap yourself, there are lots of tools out there who can help you generate your XML sitemap.

Sitemaps won’t really help you rank better, at least not directly. But they help search engines find your site and all your URLs, and that makes it easier for you to climb the rankings.

Speed this process up even more by submitting your sitemap directly through Google Search Console. Go to Sitemaps under Crawl, and click Add/Test Sitemap. You can do the same with Bing’s Webmaster Tools. Use these tools to check for any errors in your sitemap that could impede the indexing of your site.

Step 3: Off Page SEO

You can always take the direct route and submit your site’s URL directly to the search engines. Submitting your site to Google is simple: visit their page, enter your URL, complete the Captcha to prove you’re human and click Submit Request. You can also go through Search Console if you have an account. You can submit your site to Bing using their Webmaster Tools, which requires an account. Use crawl errors to find issues that might be blocking crawlers.

Make it easier for search engine spiders to find your site by getting your URL out there. Put a link to your website in your social media pages. This won’t really help you rank better in search results, but Google does crawl and index social media pages, so it will still see your links. It’s extra important that you have a Google+ account so you can use the rel=”publisher” tag to get your company information to appear in Google’s rich snippets.

If you’ve got a YouTube account, post a short video explaining the features of your new site and add a link in the video description. If you’re on Pinterest, pin a high-resolution screenshot with your URL and a description (remember to use your keywords in the description).

This last part of off-page SEO can be a bit tricky: Submitting your URL to a Web directory. Web directories were once a common way for SEOs to generate easy backlinks for their sites. The issue, however, is that many of these sites had a lot of spam content and provided little value to users. So submitting your URL to several low quality directories could do more harm than good.

Do a little bit of homework on blog directories to find directories with high authority. Some directories to consider are: DMOZ, one of the top web directories and Alltop, if you’ve got an RSS feed and a decent amount of content. Also check out trusted resources online for curated lists of credible directories.

In Conclusion

Getting found online is the end goal of your SEO. But before people can find you, the search engines need to first. You can always publish your site, sit back, relax and wait for them to come to you, but that won’t get you the best results. Use the methods listed above to improve the crawling and indexing of your pages so you can start ranking faster and build your audience.

What steps have you taken to get your pages crawled and indexed by search engines? What on page and off-page challenges have you encountered?

  • MizAkita

    Pretty basic stuff but still works in the world of SEO

Recommended

Learn Coding Online
Learn Web Development

Start learning web development and design for free with SitePoint Premium!

Instant Website Review

Use Woorank to analyze and optimize your website to improve your website to improve your ranking!

Run a review to see how your site can improve across 70+ metrics!

Get the latest in Front-end, once a week, for free.