JavaScript
Article
By Craig Buckler

Retrofit Your Website as a Progressive Web App

By Craig Buckler

Woman in a 1950s, retro scene looking at a tablet with Progressive Web App icons

There’s been a lot of buzz around Progressive Web Apps (PWAs) lately, with many people questioning whether they represent the future of the (mobile) web. I’m not going to get into the whole native app vs PWA debate, but one thing is for sure — they go a long way to enhancing mobile and improving its user experience. With mobile web access destined to surpass that of all other devices combined by 2018, can you afford to ignore this trend?

The good news is that making a PWA is not hard. In fact, it’s quite possible to take an existing website and convert it into a PWA. And that’s exactly what I’ll be doing in this tutorial — by the time you’re finished, you’ll have a website that behaves like a native web app. It will work offline and have it’s own home screen icon.

What Are Progressive Web Apps?

Progressive Web Apps (referred to as “PWAs”) are an exciting innovation in web technology. PWAs comprise a mixture of technologies to make a web app function like a native mobile app. The benefits for developers and users overcome the constraints imposed by web-only and native-only solutions:

  1. You only need one app developed with open, standard W3C web technologies. There’s no need to develop separate native codebases.
  2. Users can discover and try your app before installation.
  3. There’s no need to use an AppStore, abide with arcane rules or pay fees. Application updates occur automatically without user interaction.
  4. Users are prompted to “install” which adds an icon to their home screen.
  5. When launched, the PWA displays an attractive splash screen.
  6. The browser chrome options can be modified if necessary to provide a full-screen experience.
  7. Essential files are cached locally so PWAs respond faster than standard web apps (they can even be faster than native apps).
  8. Installation is lightweight – perhaps a few hundred KB of cached data.
  9. All data exchanges must occur over a secure HTTPS connection.
  10. PWAs function offline and can synchronize data when the connection returns.

It’s early days, but case studies are positive. Flipkart, India’s largest e-commerce site, experienced a 70% increase in sales conversions and trebled on-site time when they abandoned their native app for a PWA. Alibaba, the world’s largest business trading platform, experienced a similar conversion rate increase of 76%.

Solid PWA technology support is available in Firefox, Chrome and the other Blink-based browsers. Microsoft is working on an Edge implementation. Apple remains silent although there are promising comments in the WebKit five-year plan. Fortunately, browser support is mostly irrelevant…

Progressive Web Apps are Progressive Enhancements

Your app will still run in browsers which don’t support PWA technology. The user won’t get the benefits of offline functionality but everything will continue to work as before. Given the cost-to-benefit rewards, there’s little reason not to add PWA technologies to your system.

It’s Not Just Apps

Google has led the PWA movement so most tutorials describe how to build a Chrome-based native-looking mobile app from the ground-up. However, you don’t need a special single-page app or have to follow material interface design guidelines. Most websites can be PWA-ized within a few hours. That includes your WordPress or static site. Smashing Magazine announced they were running as a PWA while this article was being written!

Demonstration Code

Demonstration code is available from https://github.com/sitepoint-editors/pwa-retrofit

It provides a simple four-page website with a few images, one stylesheet and a single main JavaScript file. The site works in all modern browsers (IE10+). If the browser supports PWA technologies the user can read previously-viewed pages when they’re offline.

To run the code, ensure Node.js is installed then start the provided web server in your terminal with:

node ./server.js [port]

where [port] is optional and defaults to 8888. Open Chrome or another Blink-based browser such as Opera or Vivaldi then navigate to http://localhost:8888/ (or whichever port you specified). You can also open the Developer Tools (F12 or Cmd/Ctrl + Shift + I) to view various console messages.

open website in browser

View the home page, and perhaps one other, then go offline by either:

  1. Stopping the web server with Cmd/Ctrl + C, or
  2. Check the Offline checkbox in the Network or Application – Service Workers tab of the Developer Tools.

Revisit any of the pages you viewed earlier and they will still load. Visit a page you’ve not seen to be presented with a “you’re offline” page containing a list of viewable pages:

offline pages

Connect a Device

You can also view the demonstration page on an Android smartphone connected to your PC/Mac via USB. Open the Remote devices panel from More tools in the top-left three-dot menu.

remote devices

Select Settings on the left and click Add Rule to forward port 8888 to localhost:8888. You can now open Chrome on the smartphone and navigate to http://localhost:8888/.

You can use the browser menu to “Add to Home screen”. Make a couple of visits and the browser should prompt you to “install”. Both options create a new icon on your home screen. Browse a few pages then close Chrome and disconnect your device. You can then launch the PWA Website app – you’ll see a splash screen and be able to view pages you read previously despite having no connection to the server.

There are three essential steps to transform your website into a Progressive Web App…

Step 1: Enable HTTPS

PWAs require an HTTPS connection for reasons which will become apparent shortly. Prices and processes will differ across hosts but it’s worth the cost and effort, given that Google search is ranking secure sites higher.

HTTPS is not necessary for the demonstration above because Chrome permits the use of localhost or any 127.x.x.x address for testing. You can also test PWA technology on HTTP sites if you launch Chrome with the following command line flags:

  • --user-data-dir
  • --unsafety-treat-insecure-origin-as-secure

Step 2: Create a Web App Manifest

The web app manifest provides information about the application such as the name, description, and images which are used by the OS to configure home screen icons, splash pages and the viewport. In essence, the manifest is a single file alternative to the numerous vendor-specific icon and theme meta tags you may already have in your pages.

The manifest is a JSON text file in the root of your app. It must be served with a Content-Type: application/manifest+json or Content-Type: application/json HTTP header. The file can be called anything but has been named /manifest.json in the demonstration code:

{
  "name"              : "PWA Website",
  "short_name"        : "PWA",
  "description"       : "An example PWA website",
  "start_url"         : "/",
  "display"           : "standalone",
  "orientation"       : "any",
  "background_color"  : "#ACE",
  "theme_color"       : "#ACE",
  "icons": [
    {
      "src"           : "/images/logo/logo072.png",
      "sizes"         : "72x72",
      "type"          : "image/png"
    },
    {
      "src"           : "/images/logo/logo152.png",
      "sizes"         : "152x152",
      "type"          : "image/png"
    },
    {
      "src"           : "/images/logo/logo192.png",
      "sizes"         : "192x192",
      "type"          : "image/png"
    },
    {
      "src"           : "/images/logo/logo256.png",
      "sizes"         : "256x256",
      "type"          : "image/png"
    },
    {
      "src"           : "/images/logo/logo512.png",
      "sizes"         : "512x512",
      "type"          : "image/png"
    }
  ]
}

A link to this file is required in the <head> of all your pages:

<link rel="manifest" href="/manifest.json">

The main manifest properties are:

  • name – the full name of the application to be displayed to the user
  • short_name – the short name for situations where there is insufficient space for the full name
  • description – a long description of the application
  • start_url – the relative URL to start the application (typically /)
  • scope – the navigation scope. For example, a scope of /app/ would restrict the app to that folder
  • background-color – the background color used for splash screens and browser chrome (if required)
  • theme_color – the application’s colour, typically the same as the background, which can affect how the app is displayed
  • orientation – the preferred orientation: any, natural, landscape, landscape-primary, landscape-secondary, portrait, portrait-primary, and portrait-secondary
  • display – the preferred view: fullscreen (no chrome), standalone (looks like a native app), minimal-ui (a small set of UI controls) and browser (a conventional browser tab)
  • icons – an array of image objects defining the src URL, sizes and type. A range of icons should be defined.

MDN provides a full list of Web App Manifest properties.

The Manifest section of Chrome’s Development Tools Application tab validates your manifest JSON and provides an “Add to homescreen” link which functions on desktop devices:

PWA manifest

Step 3: Create a Service Worker

Service Workers are programmable proxies which can intercept and respond to network requests. They are a single JavaScript file which resides in the application root.

Your page JavaScript (/js/main.js in the demonstration code) can check for service worker support and register the file:

if ('serviceWorker' in navigator) {

  // register service worker
  navigator.serviceWorker.register('/service-worker.js');

}

If you don’t need offline capabilities, simply create an empty /service-worker.js file – users will be prompted to install your app!

Service workers can be bewildering but you should be able to adapt the demonstration code for your own purposes. It is a standard web worker script which the browser downloads (when possible) and runs on a separate thread. It has no access to the DOM or other page APIs but will intercept network requests triggered by page changes, asset downloads, and Ajax calls.

This is the primary reason your site requires HTTPS. Imagine the chaos if a third-party script could inject its own service worker from another domain. It would be able to examine and modify all data exchanges between the client and server!

Service workers react to three primary events: install, activate and fetch.

Install Event

This occurs when the application is installed. It is typically used to cache essential files using the Cache API.

First, we’ll define some configuration variables for:

  1. The cache name (CACHE) and version (version). Your application can have multiple cache stores but we only require one. A version number is applied, so if we make significant changes a new cache will be used and all previously cached files are ignored.
  2. An offline page URL (offlineURL). This is a page which will be presented when the user is offline and attempts to load a page they have not visited before.
  3. An array of essential files to install which ensure the site functions offline (installFilesEssential). This should include assets such as CSS and JavaScript but I’ve also included the home page (/) and logo. You should also include variations such as / and /index.html if URLs can be addressed in more than one way. Note that offlineURL is added to this array.
  4. Optionally, an array of desirable files (installFilesDesirable). These will be downloaded if possible but will not make the installation abort on failure.
// configuration
const
  version = '1.0.0',
  CACHE = version + '::PWAsite',
  offlineURL = '/offline/',
  installFilesEssential = [
    '/',
    '/manifest.json',
    '/css/styles.css',
    '/js/main.js',
    '/js/offlinepage.js',
    '/images/logo/logo152.png'
  ].concat(offlineURL),
  installFilesDesirable = [
    '/favicon.ico',
    '/images/logo/logo016.png',
    '/images/hero/power-pv.jpg',
    '/images/hero/power-lo.jpg',
    '/images/hero/power-hi.jpg'
  ];

The installStaticFiles() function adds files to the cache using the promise-based Cache API. A return value is only generated when the essential files are cached:

// install static assets
function installStaticFiles() {

  return caches.open(CACHE)
    .then(cache => {

      // cache desirable files
      cache.addAll(installFilesDesirable);

      // cache essential files
      return cache.addAll(installFilesEssential);

    });

}

Finally, we add an install event listener. The waitUntil method ensures the service worker will not install until all enclosed code has executed. It runs installStaticFiles() then self.skipWaiting() to make the service worker active:

// application installation
self.addEventListener('install', event => {

  console.log('service worker: install');

  // cache core files
  event.waitUntil(
    installStaticFiles()
    .then(() => self.skipWaiting())
  );

});

Activate Event

This occurs when the service worker is activated either immediately after installation or on return. You may not require this handler but the demonstration code uses one to delete old caches when they exist:

// clear old caches
function clearOldCaches() {

  return caches.keys()
    .then(keylist => {

      return Promise.all(
        keylist
          .filter(key => key !== CACHE)
          .map(key => caches.delete(key))
      );

    });

}

// application activated
self.addEventListener('activate', event => {

  console.log('service worker: activate');

    // delete old caches
  event.waitUntil(
    clearOldCaches()
    .then(() => self.clients.claim())
    );

});

Note the final self.clients.claim() call sets this service worker as the active worker for the site.

Fetch Event

This occurs whenever a network request is made. It calls the respondWith() method to hijack GET requests and return:

  1. An asset from the cache.
  2. If #1 fails, the asset is loaded from the network using the Fetch API) (unrelated to the service worker fetch event). That asset is then added to the cache.
  3. If #1 and #2 fail, an appropriate response is returned.
// application fetch network data
self.addEventListener('fetch', event => {

  // abandon non-GET requests
  if (event.request.method !== 'GET') return;

  let url = event.request.url;

  event.respondWith(

    caches.open(CACHE)
      .then(cache => {

        return cache.match(event.request)
          .then(response => {

            if (response) {
              // return cached file
              console.log('cache fetch: ' + url);
              return response;
            }

            // make network request
            return fetch(event.request)
              .then(newreq => {

                console.log('network fetch: ' + url);
                if (newreq.ok) cache.put(event.request, newreq.clone());
                return newreq;

              })
              // app is offline
              .catch(() => offlineAsset(url));

          });

      })

  );

});

The final call to offlineAsset(url) returns an appropriate response using a couple of helper functions:

// is image URL?
let iExt = ['png', 'jpg', 'jpeg', 'gif', 'webp', 'bmp'].map(f => '.' + f);
function isImage(url) {

  return iExt.reduce((ret, ext) => ret || url.endsWith(ext), false);

}


// return offline asset
function offlineAsset(url) {

  if (isImage(url)) {

    // return image
    return new Response(
      '<svg role="img" viewBox="0 0 400 300" xmlns="http://www.w3.org/2000/svg"><title>offline</title><path d="M0 0h400v300H0z" fill="#eee" /><text x="200" y="150" text-anchor="middle" dominant-baseline="middle" font-family="sans-serif" font-size="50" fill="#ccc">offline</text></svg>',
      { headers: {
        'Content-Type': 'image/svg+xml',
        'Cache-Control': 'no-store'
      }}
    );

  }
  else {

    // return page
    return caches.match(offlineURL);

  }

}

The offlineAsset() function checks whether the request is for an image and returns an SVG containing the text “offline”. All other requests return the offlineURL page.

The Service Worker section of Chrome’s Development Tools Application tab provides information about your workers, with errors and facilities to force reload and make the browser go offline:

PWA service workers

The Cache Storage section lists all caches within the current scope and the cached assets they contain. You may need to click the refresh button at the bottom of the pane if when the cache is updated:

PWA cached files

Unsurprisingly, the Clear storage section can delete your service worker and caches:

PWA clear cached files

Bonus Step 4: Create a Useful Offline Page

The offline page can be static HTML informing the user that the page they requested is not available offline. However, we can also provide a list of page URLs which are available to read.

The Cache API can be accessed within our main.js script. However, the API uses promises which fail in unsupported browsers and will cause all JavaScript to halt execution. To avoid this, we’ll add code which checks whether the offline list element and the Caches API is available before loading another /js/offlinepage.js JavaScript file (which must be present in the installFilesEssential array above):

// load script to populate offline page list
if (document.getElementById('cachedpagelist') && 'caches' in window) {
  var scr = document.createElement('script');
  scr.src = '/js/offlinepage.js';
  scr.async = 1;
  document.head.appendChild(scr);
}

/js/offlinepage.js locates the most recent cache by version name, gets a list of all URL keys, removes non-page URLs, sorts the list and appends it to the DOM node with the ID cachedpagelist:

// cache name
const
  CACHE = '::PWAsite',
  offlineURL = '/offline/',
  list = document.getElementById('cachedpagelist');

// fetch all caches
window.caches.keys()
  .then(cacheList => {

    // find caches by and order by most recent
    cacheList = cacheList
      .filter(cName => cName.includes(CACHE))
      .sort((a, b) => a - b);

    // open first cache
    caches.open(cacheList[0])
      .then(cache => {

        // fetch cached pages
        cache.keys()
          .then(reqList => {

            let frag = document.createDocumentFragment();

            reqList
              .map(req => req.url)
              .filter(req => (req.endsWith('/') || req.endsWith('.html')) && !req.endsWith(offlineURL))
              .sort()
              .forEach(req => {
                let
                  li = document.createElement('li'),
                  a = li.appendChild(document.createElement('a'));
                  a.setAttribute('href', req);
                  a.textContent = a.pathname;
                  frag.appendChild(li);
              });

            if (list) list.appendChild(frag);

          });

      })

  });

Development Tools

If you think JavaScript debugging is tough, service workers won’t be much fun! Chrome’s Application tab of the Developer Tools provides a solid set of features and logging statements are also output to the console.

You should consider running your app in an Incognito window during development since cached files are not retained after you close the tab.

Firefox offers a JavaScript debugger accessed from the Service Workers option of the tools menu. Better facilities are promised soon.

Finally, the Lighthouse extension for Chrome also provides useful information about your PWA’s implementation.

PWA Gotchas

Progressive Web Apps require new technologies so some caution is advised. That said, they are an enhancement of your existing website which should take no longer than a few hours and have no negative effect on unsupported browsers.

Developer opinions vary but there are several points to consider…

URL Hiding

The demonstration site hides the URL bar which I would not recommend unless you have a single-URL app such as a game. The manifest options display: minimal-ui or display: browser are possibly best for most sites.

Cache Overload

You could cache every page and asset on your site. That’s fine for small sites but would it be practical for those with thousands of pages? No one is likely to be interested in all your content and device storage limits could be exceeded. Even if you only store visited pages and assets like the demonstration, the cache could grow excessively.

Perhaps consider:

  • only caching important pages such as the home, contact, and the most recent articles
  • not caching images, videos and other large files
  • regularly wiping older cached files
  • providing a “store this page for offline reading” button so the user can choose what to cache.

Cache Refreshing

The demonstration looks for assets in the cache before loading from the network. That’s great when users are offline but means they could be viewing old pages even when they’re online.

URLs for assets such as images and videos should never change so long-term caching is rarely a problem. You can ensure they remain cached for at least a year (31,536,000 seconds) with the Cache-Control HTTP header:

Cache-Control: max-age=31536000

Pages, CSS and script files can change more frequently so you could set a shorter expiry of 24 hours and ensure it is validated against the server version when online:

Cache-Control: must-revalidate, max-age=86400

You could also consider cache-busting techniques to ensure older assets cannot be used, e.g. naming your CSS file styles-abc123.css and changing the hash on every release.

Caching can become complex so I’d recommend you read Jake Archibold’s Caching best practices & max-age gotchas.

The following resources are useful if you want to know more about Progressive Web Apps:

There are also many articles online which influenced how I approached this demonstration code. Feel free to adapt the code and let me know it went in the comments. Best of luck!

This article was peer reviewed by AJ Latour, Panayiotis «pvgr» Velisarakos and Dave Maxwell. Thanks to all of SitePoint’s peer reviewers for making SitePoint content the best it can be!

  • Craig Buckler

    I hope it never becomes a major ranking factor!
    I wouldn’t want every site owner to think that caching all their pages on my smartphone is a good idea!

    • M⃠ ⃠S⃠ ⃠i⃠ ⃠N⃠ ⃠L⃠u⃠n⃠d⃠

      Yeah.
      I have a bad feeling about this….

      • Craig Buckler

        Any technology can be abused! PWAs are great and browsers already have some rudimentary management tools if you don’t want a site to store data locally.

  • M⃠ ⃠S⃠ ⃠i⃠ ⃠N⃠ ⃠L⃠u⃠n⃠d⃠

    Any online example of such a site?
    I’m definitely not installing node to test it.

    Something actually useful, and not one big collection of .

    I especially want to see if they have solved the old problems with “Installed” websites, that have prevented us to use it before.

    For example there used so be no way to handle multiple tabs and deep links, or URLs, which made the site work like one of those useless framaset-sites from the 90’s.
    Basically it took away all the good parts of using a website instead of a shitty phone-app

    • Craig Buckler

      You don’t need Node.js. The server.js application is provided for those who don’t have a development server. If you have Apache, nginx or similar installed, you can copy the files and configure as necessary. Just make sure the files are served from a root folder, i.e. http://localhost/ and not http://localhost/sub-folder/. Installing Node.js is a far easier option!

      Several sites have PWA-ized such as https://www.smashingmagazine.com/ mentioned above. Jeremy Keith’s excellent https://resilientwebdesign.com/ caches the full 16Mb site so it can be read offline. (Admittedly, 16Mb could be considered a little too much, but less useful native apps are often considerably larger).

      The level of “installation” and browser interaction is up to you. You could simply enable home page icons. Or you could have a site which caches regularly-used assets but still appears in a browser UI. Or you could have a full-screen app/game which is indistinguishable from a native alternative.

      • M⃠ ⃠S⃠ ⃠i⃠ ⃠N⃠ ⃠L⃠u⃠n⃠d⃠

        Ok, thanks tested.

        https://www.smashingmagazin

        …definitely doesn’t work offline.

        Not sure what if anything Im supposed to do as a visitor.

        https://resilientwebdesign
        …seems to work.
        But then it only seem to have a couple of huge lines of text on each page.
        So its probably just one normal page spread out anyway.

  • Regiane Folter

    That’s a very interesting article! And that’s true, the Progressive Apps are becoming a must! If you want to know the journey of a developer that created a PWA recently, I recommend this Blog Post: https://blog.uruit.com/2017/03/22/coding-progressive-app-development/

  • Careful with service workers, have it but dont cache anything, at least you are 1000% sure that this should be available offline and it wont change for years—you sure you have these files? I’m sure no-one does.

    rhcloud hosted sites wont’t get service workers to work: SW need application/javascript MIME, which they won’t provide.

    Use manup.js from github—it will help you to make your webapp installable as webapp, at least from Chrome and Edge browsers.

    Don’t grow on that PWA stuff, nor you shoudn’t get mad over AMP. It’s all catchy, but what’s behind needs your datailed research.

    So what you should do? Make you site/webapp responsive and UX tested. This should actually be the message.

Recommended
Sponsors
Get the latest in JavaScript, once a week, for free.