Using cache in multi-stage builds with GitLab CI & Docker
From this article you will learn: - Why caching does not work with multi-stage builds while building images in gitlab CI - How to solve this problem and build them 4x faster
Example will be provided on below Dockerfile:
FROM composer:2 as vendor WORKDIR /app COPY --chown=www-data:www-data app/composer.json composer.json COPY --chown=www-data:www-data app/composer.lock composer.lock RUN composer install --no-dev --optimize-autoloader --quiet --no-scripts FROM node:15.1 as frontend WORKDIR /front COPY app/ /front/ RUN yarn install --silent && yarn build --no-progress FROM php:7.4-fpm # install deps and other needed tools COPY --from=frontend /front/ /application/ COPY --from=vendor /app/vendor/ /application/vendor/
Gitlab build stage (for sake of simplicity I removed configuration which is needed to run it properly, but not relevant to problem we are trying to solve).
variables: DEV_IMAGE: "php-app:$CI_COMMIT_SHORT_SHA" image_build: stage: build image: docker:19.03.8 script: - docker pull $DEV_IMAGE || true - docker build --cache-from $DEV_IMAGE --tag $DEV_IMAGE -f . - docker push $DEV_IMAGE
'|| true' ensures the build will pass the first time you run it, hence there will be error thrown while trying to pull image that do not yet exists.
Ok, so we have ready to go pipeline & build. Unfortunately, despite using --cache-from
flag, the image is rebuilt from scratch, even with no changes to Dockerfile
at all. That means caching configuration provided above is useless, you will need to rebuild everything from scratch every time and will take looong time: Duration: 5 minutes 59 seconds
The solution
Simple answer to above problem is: push intermediate stages to registry and reuse them in the following stages.
variables: VENDOR_STAGE_IMAGE: "php-app-vendor:$CI_COMMIT_SHORT_SHA" FRONTEND_STAGE_IMAGE: "php-app-frontend:$CI_COMMIT_SHORT_SHA" FINAL_IMAGE: "php-app:$CI_COMMIT_SHORT_SHA" image_build: stage: build image: docker:19.03.8 script: # needs to push intermediate images separately in order to use cache # vendor stage - docker pull ${VENDOR_STAGE_IMAGE} || true - docker build --target vendor --cache-from ${VENDOR_STAGE_IMAGE} --tag ${VENDOR_STAGE_IMAGE} -f . - docker push ${VENDOR_STAGE_IMAGE} # frontend stage - docker pull ${FRONTEND_STAGE_IMAGE} || true - docker build --target frontend --cache-from ${FRONTEND_STAGE_IMAGE} --cache-from ${VENDOR_STAGE_IMAGE} --tag ${FRONTEND_STAGE_IMAGE} -f . - docker push ${FRONTEND_STAGE_IMAGE} # final stage - docker pull ${FINAL_IMAGE} || true - docker build --cache-from ${FINAL_IMAGE} --cache-from ${FRONTEND_STAGE_IMAGE} --cache-from ${VENDOR_STAGE_IMAGE} --tag ${FINAL_IMAGE} -f . - docker push ${FINAL_IMAGE}
Despite frontend stage does not use vendor stage to build itself, remember to put the --cache-from vendor in the frontend build command. Multi-stage build goes from top to bottom and builds every stage, no matter if previous stages are needed for desired one.
Finally, after applying above changes, cache is used and the build time drops significantly: Duration: 1 minute 40 seconds