How to Optimize Docker-based CI Runners with Shared Package Caches

November 21, 2017

At Unleashed Technologies we use Gitlab CI with Docker runners for our continuous integration testing. We’ve put significant effort into speeding up the build execution speeds. One of the optimizations we made was to share a cache volume across all the CI jobs, allowing them to share files like package download caches.

Configuring the Docker runner was really simple — we simply dropped volumes = ["/srv/cache:/cache:rw"] into our config.toml file:

concurrent = 6
check_interval = 0

[[runners]]
  name = "ut-ci01"
  url = "https://gitlab.example.com/"
  token = "xxxxxxxxxxxxx"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "unleashed/php:7.1"
    privileged = false
    disable_cache = false
    volumes = ["/srv/cache:/cache:rw"]
  [runners.cache]

As a result, all CI jobs will have a /cache directory available (which is mapped to /srv/cache on the Docker host).

The next step was making the package managers utilize this cache directory whenever jobs run commands like composer install or yarn install. Luckily, these package managers allow us to configure their cache directories using environment variables:

Composer: COMPOSER_CACHE_DIR
Yarn: YARN_CACHE_FOLDER
npm: NPM_CONFIG_CACHE
bower: bower_storage__packages
RubyGems: GEM_SPEC_CACHE
pip: PIP_DOWNLOAD_CACHE

So we simply added these ENV directives in the Dockerfiles for our base images:

ENV COMPOSER_CACHE_DIR /cache/composer
ENV YARN_CACHE_FOLDER /cache/yarn
ENV NPM_CONFIG_CACHE /cache/npm
ENV bower_storage__packages /cache/bower
ENV GEM_SPEC_CACHE /cache/gem
ENV PIP_DOWNLOAD_CACHE /cache/pip

Now, whenever a job needs a package installed, it’ll pull from our local cache instead of downloading from a remote server! This provides a noticeable speed improvement for our builds.

This quick tip was originally published on Colin’s blog, and republished here with the author’s permission.