How to Run an SEO Test

This article is part of an SEO series from WooRank. Thank you for supporting the partners who make SitePoint possible.

As with articles on most other digital marketing channels, a great many SEO guides, checklists and how-tos include a line, paragraph or section about testing. Make sure you test any changes you make to your website. Constantly test various optimizations to find the ones with the biggest impacts. Whatever you’re doing, testing is an integral part of digital marketing. But what exactly does it mean to “test” something for SEO?

Fortunately for other marketers, paid marketing is pretty straightforward to test: Split test an ad, landing page or offer and draw a direct line between change and ROI improvements. Unfortunately for SEOs looking to increase search rankings, search engines operate as black boxes, and rank sites relative to each other and the search term. Your ranking may have gone up, but was it because of your new title tag, or did your competitor get caught paying for links and penalized?

Lucky for you, if you follow this four point guide to SEO tests, you’ll be able to isolate variables, measure the effectiveness of changes and optimize for increased search rankings.

1. Choose the Right Test Subjects

It’s important to start off any SEO test by picking the right test subjects. First of all, you don’t want to go around messing with your most important pages: your homepage and pages that already rank well for highly competitive keywords that get lots of volume. If you end up making a change that isn’t exactly an improvement, it could be difficult for you to recover your lost rankings.

Start by going back to your SEO strategy and keyword research to make sure your keywords are targeting the same types of user with the same, or very similar, search intent.

Your best keywords to use in tests generally rank in the top 20 or 30 search results. Anything lower than 30 tends to be too volatile to get a good data set. You could jump several rankings one day, only to lose them the next. This will result in very noisy data. You should also avoid testing with pages that rank in the top seven or eight results for their keywords. The reason being that it can be very difficult to cause any movement in the top five results by changing just one thing at a time. Most of the top results have also acquired a lot of links, which you can’t control, that could outweigh any on page optimizations you make.

Ranking stability isn’t the only selection criteria when it comes to test subjects. You also need to take stock of the test environment: The SERP. Check your Google Search Console Search Analytics and find a keyword that has a stable search ranking over time.

Google Search Console stable ranking

Avoid using keywords with landing page rankings that vary widely, or have unpredictable search volume.

Google Search Console volatile ranking

Sadly, Google recently throttled search volume data in its Keyword Planner tool. But there’s good news: You can still find search volumes in the WooRank SERP Checker tool, on top of tracking your keywords over time.

If you don’t have a WooRank account, you can carry out the process manually: Using Ad Preview and Diagnosis set your target location, language and Google domain, then do an actual search using your target keyword and make a note of the first page results. This prevents your location or search history from affecting the results. This tool often only provides the first 50 results, so another free tool is isearchfrom. Do this for a couple of days, or weeks if you’re really thorough, and keep track of pages displayed in the results. SERPs with a lot of volatility generally mean there’s outside forces at work you can’t control or isolate. While you’re doing this, check for any featured snippets. The presence of a Google Answer Box in SERPs will throw off your results.

Google Search Console’s Search Analytics report also allows you to track keyword positions for up to the last 90 days (or compare based on the last 7 or 28 days compared to the previous period). Use the filters to include only your keyword, then tick the Position checkbox to see how the position changed over time. You can also filter to only include queries from a specific country or device and rankings for a particular page.

Google Search Console Search Analytics

Conduct a quick link audit of your candidates and choose pages that have similar link profiles in terms of both quantity and quality. Ideally, all of your subjects for each test will have similar content, rankings, traffic and link profiles.

Why go through all this work just to find pages you’re going to tinker around with? It’s because you can’t run a true split test for SEO. In a real split test, you could create an exact copy of your landing page, change one thing and then let it run. But you can’t do that because the second page may not even get indexed due to duplicate content, much less outrank the original. Plus, trying to create an exact duplicate link profile is most likely going to blow up in your face and neither page will rank (and maybe incur a manual penalty from Google to boot).

2. Establish Test Groups

Once you’ve come up with a list of test subjects, it’s time to establish which ones will be the control and which will be for experimenting. There’s no set number of pages used in each group, but just remember that the more subjects you have, the more reliable your analysis will be.

Start by randomizing your subject pages. This is very important because, as we mentioned above, none of your pages are exactly the same. Therefore, when you assign subjects to test groups, your implicit bias could wind up skewing the groups. You could accidentally wind up with your favorite type of content, like videos, overrepresented when testing a particular tactic or optimization. The best way to randomize your test subjects is to assign each one a number or letter, and then use a randomizer to assign each page to a variable group.

Randomly assign subjects to a group

3. Run Your Test

Now it’s time to finally start making changes and measuring effects. The best way to conduct these tests is to use the two-sample t-test. In layman’s terms, the two-sample t-test is a method for calculating whether the differences between two groups are significant or due to random chance. The test process for a variable like using your keyword in the <H1> tag, for example, would look something like this:

  1. Start by measuring page ranking for the page’s neutral state for 10 days and average position each day. In this example it would probably be for pages that have keywords in the <H1> tags (because that has long been established as best practice. Remember, this is just a simplified example). The 10-day window is important: you need to give changes time to take effect, and you need a neutral baseline to compare test results against.
  2. Make your change — in this case, take the keyword out of the header tag. Wait for the new version of your page to show up in Google’s cache. To do this, do a Google search for your page, click the little green down arrow next to the URL and click Cache (or use the Google search operator It’s important you wait for the new page to get cached since this shows that Google has crawled the updated page and can therefore determine its ranking.Check Google cache

    Once the new version appears in the cache, let it run for ten days and record the average position.

  3. Add the keyword back to the <H1> tag content. Again, wait for the updated page to get cached and then let it run for 10 days and record the average position.
  4. Take the keyword back out of the <H1> tag, wait for the cache to upgrade and let it run for another 10 days, recording the daily average position in Google Search Console.
  5. Math

4. Measure and Compare

The 10 day waiting period here is important to gather a viable sample size to determine if using a keyword in the header tag improves ranking. You take out and add back in the keyword multiple times to control for any outside variables that might have affected search ranking like a sudden influx of links or a competitor falling behind you.

The math step is where you find out if your test resulted in significant changes. Use a t test calculator to calculate whether or not the difference between your two data sets is significant. Enter the data for the control group (keyword in the header tag) in one column and the data for the variable group (no keyword in the header tag) in the other column and click ‘Calculate now’.

What you want to look at in the results is the “confidence interval difference.” If the two numbers are close together, there’s not much of a significant difference. If they’re far apart the change is likely not due to random fluctuations. If the numbers are negative, that means the change resulted in lower search rankings. So, in this case, that would mean taking the keyword out of the header tag is bad for SEO.

You can do these calculations yourself but I got into content marketing to avoid doing complex mathematics.

5. Repeat

Scientific testing is all about being able to reproduce your results. SEO isn’t really a science, but we try our best. To confirm your conclusions from the first test, run the same test using a second group of pages you created back in step two. Ideally, randomizing your groups eliminated, or at least reduce the impact of, any other variables, but it never hurts to be sure. Testing your conclusion across multiple groups will verify it applies to pages with different content types, link profiles and keyword types.

You can also use this step to test the impact of different magnitudes of changes. If adding the keyword once to your header tag improves ranking, will adding it twice help even more? It probably won’t, because that would be spammy, but you could test that using one of your test groups.


Now when you’re reading all these SEO articles and you come across checklists telling you to test and retest your optimizations you know exactly what they mean. This testing process is long, and can take a lot of work for a team of one, but it will deliver data you can use to optimize your site to appear higher in search results.

Not sure where to start when it comes to what to test? No problem. Use your WooRank SEO audit criteria as a starting point.