How to Optimize Docker-based CI Runners with Shared Package Caches
At Unleashed Technologies we use Gitlab CI with Docker runners for our continuous integration testing. We’ve put significant effort into speeding up the build execution speeds. One of the optimizations we made was to share a cache volume across all the CI jobs, allowing them to share files like package download caches.
Configuring the Docker runner was really simple — we simply dropped
volumes = ["/srv/cache:/cache:rw"] into our
concurrent = 6 check_interval = 0 [[runners]] name = "ut-ci01" url = "https://gitlab.example.com/" token = "xxxxxxxxxxxxx" executor = "docker" [runners.docker] tls_verify = false image = "unleashed/php:7.1" privileged = false disable_cache = false volumes = ["/srv/cache:/cache:rw"] [runners.cache]
As a result, all CI jobs will have a
/cache directory available (which is mapped to
/srv/cache on the Docker host).
The next step was making the package managers utilize this cache directory whenever jobs run commands like
composer install or
yarn install. Luckily, these package managers allow us to configure their cache directories using environment variables:
So we simply added these
ENV directives in the
Dockerfiles for our base images:
ENV COMPOSER_CACHE_DIR /cache/composer ENV YARN_CACHE_FOLDER /cache/yarn ENV NPM_CONFIG_CACHE /cache/npm ENV bower_storage__packages /cache/bower ENV GEM_SPEC_CACHE /cache/gem ENV PIP_DOWNLOAD_CACHE /cache/pip
Now, whenever a job needs a package installed, it’ll pull from our local cache instead of downloading from a remote server! This provides a noticeable speed improvement for our builds.
This quick tip was originally published on Colin’s blog, and republished here with the author’s permission.