Design & UX
Article

# Speed Index: Measuring Page Load Time a Different Way

For a long time, measuring web performance was all about minimizing the time it took for various browser events to fire, such as window.onload or the loadEventEnd navigation timing. But these timings were never really that good at capturing the actual user experience. That’s where speed index comes in.

Speed index is a relatively new method of measuring web page performance that was originally developed for WebPagetest.org. It works by measuring how much of the above-the-fold content is visually complete over time until it is 100% complete (the part that is visible to the user when loaded initially).

The reason speed index is such a good metric is because it does a far better job at measuring how the user perceives the performance of a page. Take my blog for example, I’ve spent a long time optimizing it to get the speed index as low as possible, yet most pages actually take 2-3 seconds to load fully due to MathJAX’s numerous requests and the work it does. Unless there is actual mathematical markup above-the-fold, this additional work will be completely ignored by speed index. Ignoring this makes sense from a user perception perspective because by the time the user has scrolled down to the part of the page where the math is used, it has likely loaded anyway. Optimizations for speed index also typically improve the mobile experience because they’re focused on getting something up and usable as soon as possible.

## Calculating Speed Index

To measure the speed index of a page a video capture needs to be taken of the page load. Each frame is compared against the final loaded frame to determine their percentage completeness.

Plotting the visual progress of a page over time gives a graph something like this:

The area under the curve represents the visual complete progress, but since it is unbounded it doesn’t make a good score to measure, as it will keep increasing. Instead, the speed index is the area above the curve that will approach 0 as the page gets faster.

## Measuring Visual Completeness

The simplistic approach to measure visual progress is to compare the pixel at each position with the corresponding pixel in the completed frame. This turns out to have some issues though, particularly around new elements triggering a layout that shifts other elements around. Using this method, if a photo was positioned even a single pixel off its final position, it would mark the entire sector as incomplete.

WebPagetest measures visual progress by taking a histogram of colors and calculating the difference with the final frame. While this does have issues of its own, it fixes the problem outlined above.

### Frames Per Second

WebPagetest currently captures in 10 frames per second, but that could change in the future. The more frames that are measured, the more accurate the result will be.

### Paint Events

There is another more recent method of calculating visual progress that uses browser paint events emitted via the remote debugging protocol. This removes the need to create a video recording of the page load which could potentially negatively impact the page load itself. This method is for Chromium-powered browsers only, so it cannot be applied to all browsers.

## Telemetry

The Chromium team have developed a testing framework written in Python called Telemetry that is responsible for generating all the results on the Chrome Performance Dashboard. It can also be used to run page load tests on URLs of your choosing to generate speed index and other valuable metrics like navigation timings.

Telemetry works with Chromium-powered browsers and is slowly increasing its support to browsers that implement the WebDriver API (see CR implementing IE support).

### Setting Up Chromium

The instructions to get the Chromium source are available on the Chromium website, you don’t need to run gclient sync to synchronize the submodules or build the project since we’re only interested in Telemetry which is all contained within the main repository.

### Creating a Telemetry Page Set

Telemetry works by running measurements on page sets. Measurements determine how a page is run and what metrics to gather, page sets are a collection of pages to run and the configuration under which to run them.

You will need to create a page set at ./tools/perf/page_sets/ with the URLs you are interested in measuring. Here is an example that measures two URLs on my blog:

from telemetry.page import page as page_module
from telemetry.page import page_set as page_set_module

class GrowingWithTheWebPageSet(page_set_module.PageSet):
def __init__(self):
super(GrowingWithTheWebPageSet, self).__init__(
# Forces Math.random to use the same seed to improve result consistency
make_javascript_deterministic=True,
# Sets the browser to use the generic desktop user agent
user_agent_type='desktop')

urls = [
'http://www.growingwiththeweb.com',
'http://www.growingwiththeweb.com/2014/06/fibonacci-heap.html']

# Add the pages to the page set
for url in urls:
self.AddPage(page_module.Page(page_set=self, url=url))

### Running Telemetry

Once the repository has been downloaded, the following command will run the page set:

# arguments
# use the page_cycler measurement
# use our custom page set
# use the system browser
# repeat the pageset 5 times
# allow testing of live sites (not the default)
# tell page_cycler to record and report speed index

./tools/perf/run_measurement \
page_cycler \
./tools/perf/page_sets/growing_with_the_web.py \
--browser=system \
--pageset-repeat=5 \
--use-live-sites \
--report-speed-index

### Results

This will run the measurement on your page set and report back various metrics in addition to an HTML file containing the results of this run and previous runs. Here is a snippet from the results of a run (warm means the site resources are cached):

RESULT cold_speed_index: http___www.growingwiththeweb.com= [1062,799,666] ms
Avg cold_speed_index: 842.333333ms
Sd  cold_speed_index: 201.525019ms
RESULT cold_speed_index: http___www.growingwiththeweb.com_2014_06_fibonacci-heap.html= [626,794,656] ms
Avg cold_speed_index: 692.000000ms
Sd  cold_speed_index: 89.599107ms
...
RESULT warm_speed_index: http___www.growingwiththeweb.com= [422,456] ms
Avg warm_speed_index: 439.000000ms
Sd  warm_speed_index: 24.041631ms
RESULT warm_speed_index: http___www.growingwiththeweb.com_2014_06_fibonacci-heap.html= [462,500] ms
Avg warm_speed_index: 481.000000ms
Sd  warm_speed_index: 26.870058ms

The framework is incredibly powerful, providing things like network speed emulation and recording of page sets for replay later to completely remove network noise. Explore more capabilities on the official site or by running ./tools/perf/run_measurement help list.

## How Fast is Fast Enough?

Paul Irish, a Google Chrome Developer Advocate, recommends a speed index of under 1000 and to deliver the above-the-fold content in the first 14kb of the page.

## Wrap Up

I’ve introduced the speed index metric and tools that can be used to measure them. I recommend you start tracking the speed index of your web site and other important metrics today by checking out WebPagetest and Telemetry.

• Murray

Hi Daniel

You may be interested in KaTeX, that was released this week.

http://www.intmath.com/blog/katex-a-new-way-to-display-math-on-the-web/9445

and have a demo page comparing it to MathJax here:

http://www.intmath.com/cg5/katex-mathjax-comparison.php

I found your article when looking for a timing mechanism for the comparison page.

KaTeX seems to process as fast as the loop iterations, so I used that for the timing. (There’s no documentation for KaTeX yet, so I’m unaware of signals or callback functions to determine end of processing).

For MathJax, I used the inbuilt processing completion signal to stop the timer.

Any suggestions for better ways to find the “visually complete” time would be appreciated. I’ll play with the Chromium tools one of these days.

(I’m also Australian, living in Singapore.)

Regards
Murray

• http://www.growingwiththeweb.com/ Daniel Imms

Hi Murray, sorry I only just saw your comment. I actually came across KaTeX recently as well and plan on integrating it into my blog. Unfortunately I don’t think chromium is going to go ahead with a native mathml implementation, at least in the foreseeable future so we need to rely on libraries. KaTeX looks great though.

• Jam

Hi Daniel,

Nice article! I’m working on something similar, and was wondering if you created tools/perf/run_measurement. I can’t seem to find it in the chromium source. Thanks!

• http://chandlervdw.com/ Chandler Van De Water

Daniel, is there any chance you could update this for the latest version of Telemetry? I keep getting the error ImportError: cannot import name page_set. I also had to use run_benchmark but didn’t see the comments until after I figured that out… Any help would be much appreciated!

• Markus Kramer

Thanks for this great tutorial. Do you know how this should work for the current version of Telemetry?
Benchmark ‘page_cycler’ doesn’t exit. And using a similar one gives me “PermissionError: Attempted to access a file from Cloud Storage but you don’t have permission.”