For a long time, measuring web performance was all about minimizing the time it took for various browser events to fire, such as
window.onload or the
loadEventEnd navigation timing. But these timings were never really that good at capturing the actual user experience. That’s where speed index comes in.
Speed index is a relatively new method of measuring web page performance that was originally developed for WebPagetest.org. It works by measuring how much of the above-the-fold content is visually complete over time until it is 100% complete (the part that is visible to the user when loaded initially).
The reason speed index is such a good metric is because it does a far better job at measuring how the user perceives the performance of a page. Take my blog for example, I’ve spent a long time optimizing it to get the speed index as low as possible, yet most pages actually take 2-3 seconds to load fully due to MathJAX’s numerous requests and the work it does. Unless there is actual mathematical markup above-the-fold, this additional work will be completely ignored by speed index. Ignoring this makes sense from a user perception perspective because by the time the user has scrolled down to the part of the page where the math is used, it has likely loaded anyway. Optimizations for speed index also typically improve the mobile experience because they’re focused on getting something up and usable as soon as possible.
Calculating Speed Index
To measure the speed index of a page a video capture needs to be taken of the page load. Each frame is compared against the final loaded frame to determine their percentage completeness.
Plotting the visual progress of a page over time gives a graph something like this:
The area under the curve represents the visual complete progress, but since it is unbounded it doesn’t make a good score to measure, as it will keep increasing. Instead, the speed index is the area above the curve that will approach 0 as the page gets faster.
Measuring Visual Completeness
The simplistic approach to measure visual progress is to compare the pixel at each position with the corresponding pixel in the completed frame. This turns out to have some issues though, particularly around new elements triggering a layout that shifts other elements around. Using this method, if a photo was positioned even a single pixel off its final position, it would mark the entire sector as incomplete.
WebPagetest measures visual progress by taking a histogram of colors and calculating the difference with the final frame. While this does have issues of its own, it fixes the problem outlined above.
Frames Per Second
WebPagetest currently captures in 10 frames per second, but that could change in the future. The more frames that are measured, the more accurate the result will be.
There is another more recent method of calculating visual progress that uses browser paint events emitted via the remote debugging protocol. This removes the need to create a video recording of the page load which could potentially negatively impact the page load itself. This method is for Chromium-powered browsers only, so it cannot be applied to all browsers.
The Chromium team have developed a testing framework written in Python called Telemetry that is responsible for generating all the results on the Chrome Performance Dashboard. It can also be used to run page load tests on URLs of your choosing to generate speed index and other valuable metrics like navigation timings.
Setting Up Chromium
The instructions to get the Chromium source are available on the Chromium website, you don’t need to run
gclient sync to synchronize the submodules or build the project since we’re only interested in Telemetry which is all contained within the main repository.
Creating a Telemetry Page Set
Telemetry works by running measurements on page sets. Measurements determine how a page is run and what metrics to gather, page sets are a collection of pages to run and the configuration under which to run them.
You will need to create a page set at
./tools/perf/page_sets/ with the URLs you are interested in measuring. Here is an example that measures two URLs on my blog:
Once the repository has been downloaded, the following command will run the page set:
# arguments # use the page_cycler measurement # use our custom page set # use the system browser # repeat the pageset 5 times # allow testing of live sites (not the default) # tell page_cycler to record and report speed index ./tools/perf/run_measurement \ page_cycler \ ./tools/perf/page_sets/growing_with_the_web.py \ --browser=system \ --pageset-repeat=5 \ --use-live-sites \ --report-speed-index
This will run the measurement on your page set and report back various metrics in addition to an HTML file containing the results of this run and previous runs. Here is a snippet from the results of a run (warm means the site resources are cached):
RESULT cold_speed_index: http___www.growingwiththeweb.com= [1062,799,666] ms Avg cold_speed_index: 842.333333ms Sd cold_speed_index: 201.525019ms RESULT cold_speed_index: http___www.growingwiththeweb.com_2014_06_fibonacci-heap.html= [626,794,656] ms Avg cold_speed_index: 692.000000ms Sd cold_speed_index: 89.599107ms ... RESULT warm_speed_index: http___www.growingwiththeweb.com= [422,456] ms Avg warm_speed_index: 439.000000ms Sd warm_speed_index: 24.041631ms RESULT warm_speed_index: http___www.growingwiththeweb.com_2014_06_fibonacci-heap.html= [462,500] ms Avg warm_speed_index: 481.000000ms Sd warm_speed_index: 26.870058ms
The framework is incredibly powerful, providing things like network speed emulation and recording of page sets for replay later to completely remove network noise. Explore more capabilities on the official site or by running
./tools/perf/run_measurement help list.
How Fast is Fast Enough?
I’ve introduced the speed index metric and tools that can be used to measure them. I recommend you start tracking the speed index of your web site and other important metrics today by checking out WebPagetest and Telemetry.