Rahul VatsandJarek Potiuk
4e2c764a99
* Refresh the hardened base image mirror every six hours
Docker rebuilds the hardened images in place and moves the dhi.io apt
repository on at the same time. With a weekly mirror, the base images
Airflow builds on lag the repository by up to a week, and any apt
install whose dependencies were rebuilt in between fails: build-essential
currently cannot be installed because the mirrored base has an older
libbz2 than the bzip2 the repository offers
(docker-hardened-images/catalog#689). Mirroring copies manifests between
registries without pulling layers, so running it several times a day is
cheap.
Generated-by: Claude Opus 5
* Pin the hardened base images to digests the CI has passed on
Building on the mirror's moving tags made CI non-reproducible: any
refresh of the mirror changed the image every build started from,
without a change in the repository and without the full CI having run
on it. A CI run could pass or fail depending only on when it ran, and
rebuilding the same commit later gave a different image.
The images now build on a digest, and only the upgrade check moves it,
in a PR that must pass the full CI before the new base is used. The
mirror refreshes every six hours and triggers that upgrade check when it
moves a tag past its pin, so the pins follow Docker's rebuilds - and the
dhi.io apt repository moving with them - within hours rather than drift
behind it. Every pinned and mirrored digest keeps a tag of its own, so a
pinned image never becomes untagged.
The hardened images keep stock Debian's apt sources beside dhi.io, and
apt took whichever offered the higher version, mixing Debian builds into
an image whose dependencies were built by Docker. dhi.io is now
preferred whenever it publishes the package.
Generated-by: Claude Opus 5
* Let dhi.io's package versions win over the image's own
Docker renumbered its builds ("1.0.8-6+dhi2" became
"1.0.8-6+deb13u0+dhi2", which sorts lower) and its apt repository moved
to the new numbers before its images did. Every package depending on a
renumbered build - bzip2, libffi-dev, libgpg-error-dev among them - is
then uninstallable, because apt will not replace the image's
higher-numbered copy with the repository's lower-numbered one, and the
CI image build fails (docker-hardened-images/catalog#689). Every package
in the image comes from dhi.io, so letting its repository decide their
versions is safe, and it keeps the build working through such a
renumbering.
Generated-by: Claude Opus 5
* Move the base images to Docker's current Python patch releases
Docker stops rebuilding a Python patch release once it publishes the
next, so the 3.11.17, 3.12.15 and 3.13.16 images still carry the old
package numbering that the dhi.io apt repository has moved past, and
installing packages on them fails (docker-hardened-images/catalog#689).
Docker fixed it in the images of the current patch releases.
Generated with the Python part of the upgrade check, which also pins
the new images to the digests the mirror serves for them.
Generated-by: Claude Opus 5
* Compare the registry host exactly in the mirror test
Generated-by: Claude Opus 5
* Lower the dhi.io apt pin below 1000 so dependencies can resolve
At priority 1001 the Package:* dhi.io preference forces dhi.io's build even as a downgrade, so apt keeps dhi.io's libpq5 17.11 and postgresql-client-18 (which needs libpq5 >= 18.6 from PGDG) cannot be installed. Below 1000 the preference still favours dhi.io for the packages it publishes but lets apt take a newer version from another repository to satisfy a dependency. The forced "downgrade" the >1000 priority handled no longer arises now the base image is pinned to a digest CI has passed on.
* Rank the PostgreSQL apt repository above the dhi.io preference
install_postgres adds the PGDG repository, but the hardened image's Package:* dhi.io preference outranks Debian's default, so apt kept dhi.io's libpq and postgresql-client-18 (which needs a newer libpq5 than dhi.io carries) failed with held broken packages. Pinning the PostgreSQL origin above the dhi.io preference lets apt take libpq from PGDG to satisfy the client.
* Use US spelling favors to pass the docs spellcheck
The docker-stack docs spellcheck rejects the British spelling favours in build.rst; use the US spelling there and in the matching install-script comments.
---------
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
(cherry picked from commit 48435351c4)
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Docker Image for Apache Airflow
For the ease of deployment in production, the community releases a production-ready reference container image.
The Apache Airflow community, releases Docker Images which are reference images for Apache Airflow.
Every time a new version of Airflow is released, the images are prepared in the
apache/airflow DockerHub
for all the supported Python versions.
You can find the following images there (Assuming Airflow version 3.4.0):
apache/airflow:latest- the latest released Airflow image with default Python version (3.12 currently)apache/airflow:latest-pythonX.Y- the latest released Airflow image with specific Python versionapache/airflow:3.4.0- the versioned Airflow image with default Python version (3.12 currently)apache/airflow:3.4.0-pythonX.Y- the versioned Airflow image with specific Python version
Those are "reference" regular images. They contain the most common set of extras, dependencies and providers that are often used by the users and they are good to "try-things-out" when you want to just take Airflow for a spin,
You can also use "slim" images that contain only core Airflow and are about half the size of the "regular" images but you need to add all the Reference for package extras and providers that you need separately via Building the image.
apache/airflow:slim-latest- the latest released Airflow image with default Python version (3.12 currently)apache/airflow:slim-latest-pythonX.Y- the latest released Airflow image with specific Python versionapache/airflow:slim-3.4.0- the versioned Airflow image with default Python version (3.12 currently)apache/airflow:slim-3.4.0-pythonX.Y- the versioned Airflow image with specific Python version
The Apache Airflow image provided as convenience package is optimized for size, and it provides just a bare minimal set of the extras and dependencies installed and in most cases you want to either extend or customize the image. You can see all possible extras in Reference for package extras. The set of extras used in Airflow Production image are available in the Dockerfile.
However, Airflow has more than 90 community-managed providers (installable via extras) and some of the default extras/providers installed are not used by everyone, sometimes others extras/providers are needed, sometimes (very often actually) you need to add your own custom dependencies, packages or even custom providers. You can learn how to do it in Building the image.
The production images are built in DockerHub from released version and release candidates. There are also images published from branches but they are used mainly for development and testing purpose. See Airflow Git Branching for details.
Usage
The AIRFLOW_HOME is set by default to /opt/airflow/ - this means that dags
are in default in the /opt/airflow/dags folder and logs are in the /opt/airflow/logs
The working directory is /opt/airflow by default.
If no AIRFLOW__DATABASE__SQL_ALCHEMY_CONN variable is set then SQLite database is created in
${AIRFLOW_HOME}/airflow.db.
For example commands that start Airflow see: Executing commands.
Airflow requires many components to function as it is a distributed application. You may therefore also be interested in launching Airflow in the Docker Compose environment, see: Running Airflow in Docker.
You can use this image in Helm Chart as well.