Optimised Docker builds for Rails apps

Published Monday, Apr 06 2020

Since I started using Docker and Kubernetes for my apps, I have been trying to optimise the way my images are built, so to make the builds as fast as possible while keeping the images as small as possible at the same time. In this post I would like to share some tips re: building Docker images, especially for Rails apps. We’ll see how to optimise image layer caching, as well as how to reduce the image size where possible; speaking of the caching, we’ll also see how to leverage an external storage (I use Wasabi, an affordable s3 compatible storage service) to cache the gem bundle; this way, if we make changes to the Gemfile, we won’t be able to benefit from the layer caching but we can use a bundle cached on s3 so we only need to install new gems, instead of installing everything from scratch - which can slow down our builds considerably; in fact, downloading a compressed bundle from s3 and using that to only install new gems, is typically a lot faster than reinstalling the whole bundle. Using the same trick, we will also cache precompiled assets, so to speed up that step as well whenever we change static assets. An additional benefit of using an external storage for this caching, is that the cache is shared between multiple environments. For example, in my case I build images both on my Mac and in CI/CD with Github Actions.

For the base OS image I have been switching between Alpine and Debian/Ubuntu every now and then, but I think I finally settled on Debian-slim. It is a fact that Alpine makes for faster builds and much smaller images out of the box, however at times I’ve had to fight with incompatibilities or other issues due to musl-C being the libc of choice. Debian-slim may not have as small as footprint as Alpine, but it’s still quite small as base image and works basically with everything without hacks and workarounds, so for now I will stick with it and I suggest you do too unless you are paranoid about image size.

For this post, I will assume some familiarity with Docker and Docker builds, so I am going to skip the basics.

The dev/CI Dockerfile

For my app, I am currently using two Dockefiles: one for development and test/CI builds, and the other for production which is based on the first Dockerfile and makes use of multi stage builds. Let’s build the first Dockerfile, which we’ll call Dockerfile.dev, one step per time.

The very first lines of this Dockerfile specify Debian-slim as the base image, as well as some build args:

FROM debian:buster-slim

ARG RUBY_VERSION=2.6.5-jemalloc
ARG S3_ACCESS_KEY_ID=""
ARG S3_SECRET_ACCESS_KEY=""
ARG S3_REGION=eu-central-1
ARG S3_ENDPOINT=s3.eu-central-1.wasabisys.com
ARG BUILD_PACKAGES="build-essential yarn git-core vim tmux unzip python3-pip python3-setuptools zlib1g-dev libssl-dev"
ARG REQUIRED_PACKAGES="fullstaq-ruby-$RUBY_VERSION fullstaq-rbenv nodejs tzdata imagemagick libpq-dev"

Here we are setting the version of Ruby we want to install, as well as the s3 settings (also as build args; do NOT add these directly to the Dockerfile or these secrets will be persisted in the Docker image) that we need to download/upload the cached bundle and assets from/to s3. We are also specifying which packages are required for building stuff, and which packages are always required - this one will be identical in the prod Dockerfile. You can customise both depending on your app and how you would like your dev environment to look. One thing you may be wondering is why I am using Debian-slim instead of the official Ruby image. The reason is that I prefer using a version of Ruby that takes advantage of jemalloc to reduce the memory consumption significantly. The easiest way I found to achieve this is with Fullstaq Ruby, a version of Ruby that includes jemalloc and other optimisations that make it lighter and often faster. Fullstaq Ruby provides some packages we can install in Debian to get up and running with Ruby, so that’s what we will be doing.

One common best practice when building Docker images is placing the layers that do not change often as close to the top of the Dockerfile as possible, so for me the next few instructions are the following:

EXPOSE 3000

CMD ["bundle", "exec", "puma", "-Cconfig/puma.rb"]

ENV GEM_HOME="/home/rails/bundle"
ENV RAILS_ENV=development \
  RAILS_ROOT=/app \
  SECRET_KEY_BASE=foo \
  RAILS_SERVE_STATIC_FILES=true \
  RAILS_LOG_TO_STDOUT=true \
  BUNDLE_PATH="$GEM_HOME" \
  BUNDLE_APP_CONFIG="$GEM_HOME" \
  BUNDLE_SILENCE_ROOT_WARNING=1 \
  LANG=C.UTF-8 LC_ALL=C.UTF-8
ENV PATH $GEM_HOME/bin:$BUNDLE_PATH/gems/bin:/usr/lib/fullstaq-ruby/versions/$RUBY_VERSION/bin:${HOME}/bin:$PATH

So we expose the port 3000 which is the default for the Rails server, and set some envivonment variables that should be self explanatory. The next step is optional, and replaces the default Debian repository with a mirror of our choice, since mirrors are less frequently used than the official repo and therefore they are often faster:

RUN sed -i 's/http\:\/\/deb.debian.org/http\:\/\/ftp.de.debian.org/g' /etc/apt/sources.list

My servers are located in Germany, so I am using a German mirror. Again, this is optional. Next, we’ll add a couple of additional repositories to install Fullstaq Ruby as well as yarn and nodejs which are required for the assets precompilation:

RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends wget apt-transport-https ca-certificates curl gnupg2

RUN echo "deb https://apt.fullstaqruby.org debian-10 main" > /etc/apt/sources.list.d/fullstaq-ruby.list \
  && curl -SLfO https://raw.githubusercontent.com/fullstaq-labs/fullstaq-ruby-server-edition/master/fullstaq-ruby.asc \
  && apt-key add fullstaq-ruby.asc

RUN curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - \
  && echo 'deb http://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/yarn.list

RUN curl -sL https://deb.nodesource.com/setup_13.x | bash -

We can then install all the packages in one go to optimise layer caching:

RUN apt-get update -qq \
  && DEBIAN_FRONTEND=noninteractive apt-get upgrade -yq \
  && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends ${BUILD_PACKAGES} ${REQUIRED_PACKAGES} \
  && pip3 install awscli \
  && GOOGLE_LINUX_DL=https://dl.google.com/linux \
    && curl -sL "$GOOGLE_LINUX_DL/linux_signing_key.pub" | apt-key add - \
    && curl -sL "$GOOGLE_LINUX_DL/direct/google-chrome-stable_current_amd64.deb" \
      > /tmp/chrome.deb \
    && apt install --no-install-recommends --no-install-suggests -y \
      /tmp/chrome.deb \
    && CHROMIUM_FLAGS='--no-sandbox --disable-dev-shm-usage' \
    && sed -i '${s/$/'" $CHROMIUM_FLAGS"'/}' /opt/google/chrome/google-chrome \
    && BASE_URL=https://chromedriver.storage.googleapis.com \
    && VERSION=$(curl -sL "$BASE_URL/LATEST_RELEASE") \
    && curl -sL "$BASE_URL/$VERSION/chromedriver_linux64.zip" -o /tmp/driver.zip \
    && unzip /tmp/driver.zip \
    && chmod 755 chromedriver \
    && mv chromedriver /usr/local/bin/ \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
  && truncate -s 0 /var/log/*log

As you can see, I am installing the AWS CLI to interact with s3, and Google Chrome plus the chromedriver as well, to support system tests with Capybara; you can skip this if that’s not the case for your app. Note the last three bits of this RUN command. We are cleaning up everything concerning the apt cache in order to reduce the image size a bit.

In the next step, we set the version of Ruby we want to use with rbenv (which is used by Fullstaq Ruby) - at the moment of this writing 2.6.5 is the latest version available for Ruby 2.6 - and update both Rubygems and bundler. Then we set the work directory to the app directory /app, and create a couple of directories without which some following steps would fail.

RUN rbenv global $RUBY_VERSION && gem update --system && gem install bundler

WORKDIR $RAILS_ROOT

RUN mkdir -p ./tmp/pids && mkdir -p ./log

The next steps are designed to optimise layer caching significantly.

COPY package.json yarn.lock $RAILS_ROOT/
RUN yarn install

COPY bin/install-gems $RAILS_ROOT/bin/
COPY Gemfile Gemfile.lock $RAILS_ROOT/
RUN ./bin/install-gems

We don’t copy the whole Rails app to the image yet, because otherwise any change to any file, however small, would invalidate the cache from this point on, completely. Instead, we first copy the package.json and yarn.lock files because these files don’t change often. If they do change, we install the packages with yarn. Same things with the gem bundle: we copy a script to install the gems as well as the Gemfile; if these change, then we install the gems. You can see here that I am not executing bundle install directly; instead, I am using a script that will use s3 as cache for the bundle as explained earlier. Let’s see the contents of that script:

#!/bin/bash
set -ex

bundler_path="/home/rails/bundle"
archive="bundle.tar.gz"
s3_path="s3://mybucket/$archive"

if AWS_ACCESS_KEY_ID=$S3_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$S3_SECRET_ACCESS_KEY aws s3 --region=$S3_REGION --endpoint-url=https://$S3_ENDPOINT cp $s3_path . ; then
  tar -xzf $archive -C / && rm -f $archive
fi

bundle install -j "$(getconf _NPROCESSORS_ONLN)" --retry 3
bundle binstubs bundler --force

find $bundler_path/ -name "*.c" -delete
find $bundler_path/ -name "*.o" -delete

bundle pack --quiet

tar -zcf $archive $bundler_path

AWS_ACCESS_KEY_ID=$S3_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$S3_SECRET_ACCESS_KEY aws s3 --region=$S3_REGION --endpoint-url=https://$S3_ENDPOINT cp $archive $s3_path

rm -f $archive

In this script, we first try to download the bundle from s3 if it exists already; if it does, we extract it to the bundle directory and then run bundle install to install any new gems according to changes made to the Gemfile. We are making sure bundle can install gems with as many parallel jobs as the number of cores in the system, in order to speed up the installation. After installing the gems, we do some cleanup, and reupload the updated archive to s3, so it’s ready for use with the next build.

Note that we have installed the AWS CLI in a previous step, and the S3 settings are defined at the top of the Dockerfile as build args; we’ll see later how and where I set these variables.

Once the bundle is sorted out, it’s the turn of the assets precompilation. We copy a number of files and dirctories that are needed for the precompilation step to succeed, so we precompile the assets only if any of those files have changed.

COPY ./postcss.config.js ./tailwind.config.js ./babel.config.js $RAILS_ROOT/
COPY ./Rakefile $RAILS_ROOT/Rakefile
COPY ./app/models/application_record.rb $RAILS_ROOT/app/models/application_record.rb
COPY ./app/assets $RAILS_ROOT/app/assets
COPY ./bin $RAILS_ROOT/bin
COPY ./lib $RAILS_ROOT/lib
COPY ./app/models/constraints $RAILS_ROOT/app/models/constraints
COPY ./app/models/user.rb $RAILS_ROOT/app/models/
COPY ./app/models/*_uploader.rb $RAILS_ROOT/app/models/
COPY ./app/models/liquid $RAILS_ROOT/app/models/liquid
COPY ./config $RAILS_ROOT/config
COPY ./app/javascript $RAILS_ROOT/app/javascript

RUN ./bin/precompile-assets \
  && yarn cache clean

Of course you may have to change the files and directories according to your app; the lines above are taken from the actual Dockerfile I use for my current app. Note the yarn cache clean command: this alone can clean up 100-200MB of stuff that would bloat the final image. As you can see, we use another script here which will cache the precompiled assets to s3. This script is pretty similar to the previous:

#!/bin/bash
set -ex

assets_path="public/packs"
archive="assets.tar.gz"
s3_path="s3://mybucket/$archive"

if AWS_ACCESS_KEY_ID=$S3_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$S3_SECRET_ACCESS_KEY aws s3 --region=$S3_REGION --endpoint-url=https://$S3_ENDPOINT cp $s3_path . ; then
  tar -xzf $archive && rm -f $archive
fi

bundle exec rake webpacker:compile RAILS_ENV=production NODE_ENV=production

tar -zcf $archive $assets_path tmp/cache/webpacker/last-compilation-digest-production

AWS_ACCESS_KEY_ID=$S3_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY=$S3_SECRET_ACCESS_KEY aws s3 --region=$S3_REGION --endpoint-url=https://$S3_ENDPOINT cp $archive $s3_path

rm -f $archive

I am running the webpacker:compile rake task because I am not using the asset pipeline. If you are, you can run assets:precompile instead.

Finally, we ensure that the rest of the app is copied over:

COPY . $RAILS_ROOT

The dev Dockerfile is now ready. To use it, I have another script at bin/run that has a bunch of things, including the following:

IMAGE="user/repo"

POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
  build)
    shift

    source ~/.s3/wasabi

    VERSION=${VERSION:-"v`git rev-parse --short HEAD`"}

    export DOCKER_BUILDKIT=1

    case "$1" in
      ci)
        echo ${VERSION} > current_version.ci
        docker build -t ${IMAGE}:ci --cache-from ${IMAGE}:ci --build-arg BUILDKIT_INLINE_CACHE=1 --network=host -f Dockerfile.dev --build-arg S3_ACCESS_KEY_ID="$ACCESS_KEY" --build-arg S3_SECRET_ACCESS_KEY=$SECRET_KEY .
        shift
      ;;
      prod)
        echo ${VERSION} > current_version.prod
        docker build -t ${IMAGE}:prod --cache-from ${IMAGE}:ci --cache-from ${IMAGE}:prod --build-arg BUILDKIT_INLINE_CACHE=1 --network=host -f Dockerfile.prod --build-arg S3_ACCESS_KEY_ID="$ACCESS_KEY" --build-arg S3_SECRET_ACCESS_KEY=$SECRET_KEY .
        docker tag ${IMAGE}:prod ${IMAGE}:${VERSION}
        shift
      ;;
      *)
        echo ${VERSION} > current_version.dev
        docker build -t ${IMAGE}:dev --build-arg BUILDKIT_INLINE_CACHE=1  --network=host -f Dockerfile.dev --build-arg S3_ACCESS_KEY_ID="$ACCESS_KEY" --build-arg S3_SECRET_ACCESS_KEY=$SECRET_KEY .
        docker tag ${IMAGE}:dev ${IMAGE}:${VERSION}
      ;;
    esac

    echo $VERSION
  ;;

  push)
    ENV=dev

    shift

    if [ -n "$1" ]; then
      ENV="$1"
      shift
    fi

    VERSION=${VERSION:-`cat current_version.${ENV}`}
    docker push ${IMAGE}:${ENV}
    docker push ${IMAGE}:${VERSION}
    echo $VERSION
  ;;
  *)
    POSITIONAL+=("$1")
    shift
  ;;
esac
done
set -- "${POSITIONAL[@]}"

So we have a build command as well as a push command. With build, we first source a file outside the Git repository with the secrets for s3:

ACCESS_KEY=...
SECRET_KEY=...
REGION=eu-central-1
ENDPOINT=s3.eu-central-1.wasabisys.com

Then we set the VERSION variable to the current Git SHA if VERSION has not been set already (in CI with Github Actions, it is set by the pipeline already), and then we build the expected image depending on the parameter passed for the environment, so an image for either test/CI, or production, or dev. One important thing to note here is that we are using Buildkit since it’s faster than building images the default way. Also, we are using –cache-from to suggest Docker which existing images might be useful to speed up building; this doesn’t change anything when building locally, but in some cases where Docker is completely reset between builds (like with Github Actions), we cannot leverage previous builds directly because there are none, so we need to download existing images and use those as cache.

With this in place, we can build and push a dev CI or prod image with:

bin/run build <environment>
bin/run push <environment>

For dev images, we can just omit the environment. BTW in the run script I am using –network=host; this is because in some environments (like when a Github Actions runner runs in Kubernetes with Docker-in-Docker), there might be occasional networking issues. Long story short, using the host network for builds fixes these issues.

The prod Dockerfile

The production Dockerfile is very simple: it does a multi stage build starting from the image built for CI. For prod we don’t require any dependency that is used to build software or run tests, so the final image is much smaller. I’ll just paste the content of my Dockerfile.prod since it should be self explanatory by now:

FROM user/repo:ci AS dev

FROM debian:buster-slim

ARG RUBY_VERSION=2.6.5-jemalloc
ARG REQUIRED_PACKAGES="git-core fullstaq-ruby-$RUBY_VERSION fullstaq-rbenv nodejs tzdata imagemagick libpq-dev"

EXPOSE 3000

CMD ["bundle", "exec", "puma", "-Cconfig/puma.rb"]

ENV GEM_HOME="/home/rails/bundle"
ENV RAILS_ENV=production \
  RAILS_ROOT=/app \
  SECRET_KEY_BASE=foo \
  RAILS_SERVE_STATIC_FILES=true \
  RAILS_LOG_TO_STDOUT=true \
  BUNDLE_PATH="$GEM_HOME" \
  BUNDLE_APP_CONFIG="$GEM_HOME" \
  BUNDLE_SILENCE_ROOT_WARNING=1 \
  LANG=C.UTF-8 LC_ALL=C.UTF-8
ENV PATH $GEM_HOME/bin:$BUNDLE_PATH/gems/bin:/usr/lib/fullstaq-ruby/versions/${RUBY_VERSION}/bin:$PATH

RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends wget apt-transport-https ca-certificates curl gnupg2

RUN echo "deb https://apt.fullstaqruby.org debian-10 main" > /etc/apt/sources.list.d/fullstaq-ruby.list \
  && curl -SLfO https://raw.githubusercontent.com/fullstaq-labs/fullstaq-ruby-server-edition/master/fullstaq-ruby.asc \
  && apt-key add fullstaq-ruby.asc

RUN curl -sL https://deb.nodesource.com/setup_13.x | bash -

RUN apt-get update -qq \
  && DEBIAN_FRONTEND=noninteractive apt-get upgrade -yq \
  && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends ${REQUIRED_PACKAGES} \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
  && truncate -s 0 /var/log/*log

RUN mkdir -p $RAILS_ROOT/bin

RUN rbenv global 2.6.5-jemalloc && gem update --system && gem install bundler

WORKDIR $RAILS_ROOT

COPY --from=dev $GEM_HOME $GEM_HOME

COPY --from=dev /app/config.ru /app/Rakefile ./
COPY --from=dev /app/public ./public
COPY --from=dev /app/lib ./lib
COPY --from=dev /app/bin ./bin
COPY --from=dev /app/db ./db
COPY --from=dev /app/config ./config
COPY --from=dev /app/Gemfile* ./
COPY --from=dev /app/app ./app

RUN mkdir -p ./tmp/pids && mkdir -p ./log

As you can see we copu only what’s needed for the app to work, starting from what changes less often.

I think that’s it. This post was requested a few times so here we are. I hope it’s useful and if you have any tips on how to further optimise this, please do let me know!

Previous: Highly available, external load balancer for Kubernetes in Hetzner Cloud using haproxy and keepalived

Next: Full page caching in Rails with Nginx and Redis

Optimised Docker builds for Rails apps

The dev/CI Dockerfile

The prod Dockerfile

The author

Latest posts

Topics

License

Archives