Building a Better Web with Automated Testing on Real Devices

Share this article

Automated Testing on Real Devices

Automated Testing on Real Devices

This article was originally published on Medium.

My work is entirely dedicated to that of helping people build better, faster and more accessible apps and websites. Running Calibre has given me a lot of insight to the challenges that teams face while building and maintaining their little slices of the web.

Something that has really struck me this past year is how little we as a web industry know about the ways in which people (yep, real people, not other web developers) access the Internet, and tangentially, how antiquated our methods of delivering content to users really are.

Sure, we’ve had some major advances in the last couple of years that are dramatically improving how quickly we can push content down to devices, but ultimately as an industry not much has changed from the core premise of “load the HTML, find the other resources, then load them too.”

Dinosaurs fighting: "Eat Your Vegetablesssss!"

Left: Web developers, Right: Internet users.

Today, global Internet access is somewhere around 46.1%. That is, only half of the population on this beautiful blue marble that we call home will have a rough idea of what ‘pull to refresh’ means. The rest? Well, they’re not connected, so they probably don’t.

If you investigate the growth of the web over the last three years, it won’t take very long to find that Internet access in India has been growing at a rate of which we’ve never seen before.

In 2016 alone, India introduced 106 million people to the Internet, for the first time. To add some perspective, that’s around 290,000 people every day.

That’s growth of about 30% from 2015, and if those numbers are matched again in 2017 (this is highly likely), that’s another 140 million people.

Indeed, only 35% of Indians are online today, and the population is 1.2 billion.

We’re set for tremendous, unprecedented growth for the next few years.


Mobile usage surpassed desktop usage some time during 2014 — 51.3% of devices with an Internet connection, are hand-held. Being that hand-held devices are generally far cheaper (and often just as capable for everyday tasks as their more expensive and less portable counterparts, desktop computers) this does not come as a great surprise.

Looking for more on Jenkins and continuous integration? Check out these great links:

Check out SitePoint Premium for more books, courses and free screencasts.

Data and Connectivity

Last year I did some research to calculate how much mobile data cost in a number of locations around the globe. Taking the local minimum wage, finding the carrier with the largest market share, and finding the best value for money prepaid plan that had at least 500mb of data.

Number of hours worked for 500mb of data: India 17 hours; Brazil 13 hours; Indonesia 6 hours; Germany 1 hour

India topped out the leaderboard. More than 2 days of full time work to get a data plan that we in the western world would consider “maybe ok for your parents”.

It goes a bit further than that too, because even if you can afford to have a device, with a decent monthly data allowance, do not for one second assume that it will be fast.

Average LTE speed: Germany 20.3 Mbps; Brazil 19.68 Mbps; Indonesia 8.79 Mbps; India 6.39 Mbps

Regardless of the average LTE speed, I have more, perhaps shocking news: 60% of the worlds average mobile connections, are 2G.

That isn’t just “2G speeds”, that’s a 2G connection. On your iPhone, you might’ve seen the network advertised as ‘Edge’, and everything … well, it stopped working, right?

You may be thinking, “Well, almost everyone I know has at least a DSL connection … that should be better than the speeds I’m reading here”, and unfortunately, that isn’t quite the case.

The global average Internet connection speed? 7 Mbps. 🤕

👋 Namaste, India

Late last year, a new 4G-only wireless network provider appeared after a year of private beta made up of their friends, family and employees.

In a few short months, Jio has, in one fell swoop, changed the face of connectivity in India, as well as what I believe to be the world’s fastest customer growth curve that would bring Silicon Valley to its knees (if only they knew other countries existed 😏).

Since arrival to market, Jio have forced tariffs down by up to 80%.

Number of hours worked for 500mb of data. Before Jio: India 17, Brazil 13, Indonesia 6, Germany 1; After Jio: Brazil 13, Indonesia 6, India 4, Germany 1

The growth curve to end all growth curves

The company commercially launched its services on 5 September 2016. Within the first month of commercial operations, Jio announced that it had acquired 16 million subscribers. This is the fastest ramp-up by any mobile network operator anywhere in the world. Jio crossed the 50 million subscriber mark in 83 days of its launch. Jio crossed 100 million subscribers on 22 February, 2017. Wikipedia


50 Million subscribers in less than 90 days. 100M in less than 6 months. I can’t get over those numbers. I keep repeating them to myself.

Not Everyone Has an iPhone

Right now, you might be sitting at a nice desk, using Wi-Fi on a computer that cost somewhere in the region of $1600–3000 US dollars, or maybe a reasonably current specification Android or iPhone.

Given you’re most likely a part of the web development community (hey, that’s my audience), that isn’t surprising.

Aside from your parents, who have the same broken computer that was there when you left home … other people have current model machines, right?

Over the last few years I’ve seen and heard a lot about Android fragmentation, but I’m not certain I ever really understood it until I researched global smart phone sales.

Turns out I’d never heard of ⅔ of the phones being sold.

Global smartphone sales Q1 2017: total 379.9 million

What I find striking about these statistics is not only the tremendous number of devices (have you ever tried to picture how much space 380 million phones would take up if you stacked them in a room?), but more so the fact that I’d never heard of a handful of these vendors — who sell in excess of 200M devices in just three months. 😳

Such a wide range of devices in the market leaves plenty of web developers with a sickly feeling in their stomach. How on earth are we meant to test all these, let alone understand their quirks intimately?

A strategy that I’m fond of is:

  • Test using real hardware. (And automate your testing. Hello Calibre)
  • Test in all major browsers for a given market.
  • Test using realistic average network conditions for a given market.

There’s some subtlety in testing on the right devices, but thankfully Alex Russell was kind enough to share his 2016 research, in which he did some detective work to find a single mobile device that best represented devices in the wild, could be purchased today and would make for a trustworthy test phone in the years ahead. 🕵

The criteria for such a device is as follows:

  1. This device needed to be globally available, reasonably affordable, have minimal bloatware, and thus, a low barrier to entry.
  2. The device would need to be slightly ahead of the 2016 average so that it made for a solid candidate to purchase as a test device. (An investment!)
  3. The Nexus 5X wasn’t an option due to thermal instability. Sometimes they’re really fast, and sometimes supppper slow (A57 vs A53 Chipsets).
  4. There are billions of Android devices in the wild that are using the Qualcomm 28nm A53 Chipset. These Quad-core CPUs are everywhere and also come with hardly any cache. It’s best to choose a device on that chipset.

Most users receive phones via their carrier and the devices vary greatly over demographics and geography.

Moto G4 specs

This Phone is no slouch

After conducting this research, Alex arrived at the Motorola G4 — which, statistically, is quite a bit above average. There are a few caveats:

  • The real median is a slower, older, cheaper phone.
  • The Moto G4 is a great candidate for a device that will be the median in late 2017, early 2018. (So, think of it as an investment.)
  • There’s quite a lot of RAM (2GB) — Memory just hasn’t gotten cheap. That’s why the really cheap phones are still shipping with 512mb-1gb of RAM.

If you develop and test with a Motorola G4 starting today, and it performs ‘kinda well’, it’ll be amazing for everyone who uses newer, more powerful devices. 🎉

Device specifications are really huge factors when it comes to a smooth, delightful user experience, particularly if you’re relying on JS for interactivity and rendering. The more you put in script, the more dependent on local single-core performance your app becomes.

Lower-powered devices will feel sluggish, whereas newer devices can run a number of pages at once, without breaking a sweat.


Testing on a real device can become pretty limiting, too. Thankfully, both Web Page Test and Calibre have got you covered.

WebPageTest screen shot

WPT runs on a real Moto G4 in Dulles Virginia!

Calibre real-world and emulation targets

Calibre also has a series of real-world device and emulation targets.

Real devices are important to develop empathy and understanding, but making testing trival is crucial for teams. Just like a continuous integration process, it should be automated and run regularly to develop a full timeline of changes in performance.

In Addy Osmani’s excellent article “JavaScript Start-up Performance”, he studied the time it took to parse (that’s parse, not execute) 1mb of JavaScript on a wide array of devices.

Parsing 1mb of data

As you’ll notice, the parse time varies greatly between devices. Addy’s article is so, so important. Please go read it. 😘

Today, 620Kb of JavaScript is the global median, which sounds like a lot, but remember, this is GZipped. Forget 620kb, it’s more like 2–3mb decompressed. On a slower device, we’re talking total lock up — almost unresponsive for a number of seconds.

That’s why it’s so crucial to test with real devices, because your $1,000 iPhone performs almost as well as a reasonably new Macbook Pro. That isn’t reality.

Your Job: Building a Deep Understanding of User Experience

I hope that the information that I’ve put forward to you has given you something that you can point at, stamp your feet and say “We need to do better”, because that’s the truth. We do.

The nature of the web is change. There are no constants. Experiences vary greatly by context.

For the web development community, there are two versions of the web, the one that we use daily, and the one that we don’t.

We know ourselves that connections vary by where you are: maybe you’re the wrong end of the building, at a conference, on a plane, in a tunnel or for whatever damned reason the modem needs restarting again.

We must stop optimising for $3,000 dollar computers with fast connections.

If you take a peek outside your bubble, you’ll find that we’re really just not trying hard enough to build beautiful, fast, efficient and delightful user experiences for our beloved ‘users’.

I’m certain that no one signed up to build the wealthy western web. We all build for the World Wide Web, and it’s time that we as a community started acting this way. 👍

Make the web fast for everyone.

Huge, bellowing applause for my friends and heroes who helped me prepare this information: Karolina Szczur, Addy Osmani, Alex Russell and Bruce Lawson.

This article was originally published on Medium.

Ben SchwarzBen Schwarz
View Author

Ben is the founder of Calibre, an app for comprehensive app monitoring. He’s involved in web standards with @whatwg & @w3c, helps organize, and loves to make the web fast for everyone.

calibrepage speedperformanceRalphMTestingweb page speedwebpagetest
Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week
Loading form