nprasad2077/airflow
1
0 Fork 0
mirror of https://github.com/apache/airflow.git synced 2026-11-04 07:12:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
c8ebaa8c8e95f2ec9448e07a8f23ae63ee9d238c
airflow/docker-stack-docs/build-arg-ref.rst
T
Jarek PotiukandRahul Vats 48435351c4 Pin the hardened base images to digests the CI has passed on (#76023) 
* Refresh the hardened base image mirror every six hours

Docker rebuilds the hardened images in place and moves the dhi.io apt
repository on at the same time. With a weekly mirror, the base images
Airflow builds on lag the repository by up to a week, and any apt
install whose dependencies were rebuilt in between fails: build-essential
currently cannot be installed because the mirrored base has an older
libbz2 than the bzip2 the repository offers
(docker-hardened-images/catalog#689). Mirroring copies manifests between
registries without pulling layers, so running it several times a day is
cheap.

Generated-by: Claude Opus 5

* Pin the hardened base images to digests the CI has passed on

Building on the mirror's moving tags made CI non-reproducible: any
refresh of the mirror changed the image every build started from,
without a change in the repository and without the full CI having run
on it. A CI run could pass or fail depending only on when it ran, and
rebuilding the same commit later gave a different image.

The images now build on a digest, and only the upgrade check moves it,
in a PR that must pass the full CI before the new base is used. The
mirror refreshes every six hours and triggers that upgrade check when it
moves a tag past its pin, so the pins follow Docker's rebuilds - and the
dhi.io apt repository moving with them - within hours rather than drift
behind it. Every pinned and mirrored digest keeps a tag of its own, so a
pinned image never becomes untagged.

The hardened images keep stock Debian's apt sources beside dhi.io, and
apt took whichever offered the higher version, mixing Debian builds into
an image whose dependencies were built by Docker. dhi.io is now
preferred whenever it publishes the package.

Generated-by: Claude Opus 5

* Let dhi.io's package versions win over the image's own

Docker renumbered its builds ("1.0.8-6+dhi2" became
"1.0.8-6+deb13u0+dhi2", which sorts lower) and its apt repository moved
to the new numbers before its images did. Every package depending on a
renumbered build - bzip2, libffi-dev, libgpg-error-dev among them - is
then uninstallable, because apt will not replace the image's
higher-numbered copy with the repository's lower-numbered one, and the
CI image build fails (docker-hardened-images/catalog#689). Every package
in the image comes from dhi.io, so letting its repository decide their
versions is safe, and it keeps the build working through such a
renumbering.

Generated-by: Claude Opus 5

* Move the base images to Docker's current Python patch releases

Docker stops rebuilding a Python patch release once it publishes the
next, so the 3.11.17, 3.12.15 and 3.13.16 images still carry the old
package numbering that the dhi.io apt repository has moved past, and
installing packages on them fails (docker-hardened-images/catalog#689).
Docker fixed it in the images of the current patch releases.

Generated with the Python part of the upgrade check, which also pins
the new images to the digests the mirror serves for them.

Generated-by: Claude Opus 5

* Compare the registry host exactly in the mirror test

Generated-by: Claude Opus 5

* Lower the dhi.io apt pin below 1000 so dependencies can resolve

At priority 1001 the Package:* dhi.io preference forces dhi.io's build even as a downgrade, so apt keeps dhi.io's libpq5 17.11 and postgresql-client-18 (which needs libpq5 >= 18.6 from PGDG) cannot be installed. Below 1000 the preference still favours dhi.io for the packages it publishes but lets apt take a newer version from another repository to satisfy a dependency. The forced "downgrade" the >1000 priority handled no longer arises now the base image is pinned to a digest CI has passed on.

* Rank the PostgreSQL apt repository above the dhi.io preference

install_postgres adds the PGDG repository, but the hardened image's Package:* dhi.io preference outranks Debian's default, so apt kept dhi.io's libpq and postgresql-client-18 (which needs a newer libpq5 than dhi.io carries) failed with held broken packages. Pinning the PostgreSQL origin above the dhi.io preference lets apt take libpq from PGDG to satisfy the client.

* Use US spelling favors to pass the docs spellcheck

The docker-stack docs spellcheck rejects the British spelling favours in build.rst; use the US spelling there and in the matching install-script comments.

---------

Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
2026-10-30 13:52:53 +00:00

309 lines
28 KiB
ReStructuredText
Raw Blame History

.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
Image build arguments reference
-------------------------------
The following build arguments (``--build-arg`` in docker build command) can be used for production images.
Those arguments are used when you want to customize the image. You can see some examples of it in
:ref:`Building from PyPI packages<image-build-pypi>`.
Basic arguments
...............
Those are the most common arguments that you use when you want to build a custom image.
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| Build argument | Default value | Description |
+==========================================+===========================================+=============================================+
| ``AIRFLOW_VERSION`` | :subst-code:`|airflow-version|` | Version of Airflow. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``AIRFLOW_IMAGE_FLAVOR`` | ``hardened`` | ``hardened`` takes Python from |
| | | ``BASE_IMAGE``. ``legacy`` compiles |
| | | ``AIRFLOW_PYTHON_VERSION`` from sources on |
| | | a plain Debian ``BASE_IMAGE`` (see below |
| | | the table). |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``AIRFLOW_PYTHON_VERSION`` | | Python version recorded, as passed, in the |
| | | image labels (the published images record |
| | | the full patch release). In the hardened |
| | | flavor Python comes from ``BASE_IMAGE``: |
| | | leave it empty, and a version that does not |
| | | match ``BASE_IMAGE`` fails the build. In |
| | | the legacy flavor it is |
| | | the version compiled from sources: |
| | | ``MAJOR.MINOR`` builds the latest release |
| | | of that line. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``BASE_IMAGE`` | (see below the table) | Airflow's public mirror of the Docker |
| | | Hardened Image that Python comes from. |
| | | Point it at a ``dhi.io`` ``-fips-`` variant |
| | | to build a FIPS-compliant image. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``AIRFLOW_EXTRAS`` | (see below the table) | Default extras with which Airflow is |
| | | installed. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``ADDITIONAL_AIRFLOW_EXTRAS`` | | Optional additional extras with which |
| | | Airflow is installed. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``AIRFLOW_HOME`` | ``/opt/airflow`` | Airflow's HOME (that's where logs and |
| | | SQLite databases are stored). |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``AIRFLOW_USER_HOME_DIR`` | ``/home/airflow`` | Home directory of the Airflow user. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``AIRFLOW_PIP_VERSION`` | ``<LATEST_AVAILABLE_IN_PYPI>`` | PIP version used. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``AIRFLOW_UV_VERSION`` | ``<LATEST_AVAILABLE_IN_PYPI>`` | UV version used. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``AIRFLOW_USE_UV`` | ``false`` | Whether to use UV to build the image. |
| | | This is an experimental feature. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``ADDITIONAL_PIP_INSTALL_FLAGS`` | | additional ``pip`` flags passed to the |
| | | installation commands (except when |
| | | reinstalling ``pip`` itself) |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``PIP_PROGRESS_BAR`` | ``on`` | Progress bar for PIP installation |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``AIRFLOW_UID`` | ``50000`` | Airflow user UID. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``AIRFLOW_CONSTRAINTS`` | ``constraints`` | Type of constraints to build the image. |
| | | This can be ``constraints`` for regular |
| | | images or ``constraints-no-providers`` for |
| | | slim images. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
| ``AIRFLOW_CONSTRAINTS_REFERENCE`` | | Reference (branch or tag) from GitHub |
| | | where constraints file is taken from |
| | | It can be ``constraints-main`` or |
| | | ``constraints-3-X`` for |
| | | 3.X.* installation. In case of building |
| | | specific version you want to point it |
| | | to specific tag, for example |
| | | :subst-code:`constraints-|airflow-version|`.|
| | | Auto-detected if empty. |
+------------------------------------------+-------------------------------------------+---------------------------------------------+
.. note::
Before Airflow 2.2, the image also had ``AIRFLOW_GID`` parameter, but it did not provide any additional
functionality - only added confusion - so it has been removed.
The default ``BASE_IMAGE`` is Airflow's public mirror of the `Docker Hardened Image <https://dhi.io>`_ for
Python - for Airflow |airflow-version| that is ``ghcr.io/apache/airflow/base/python:3.13.18-debian13-dev``,
pinned to the digest the Airflow CI last passed on (see :ref:`the hardened base images <image-hardened-base>`). Pulling the mirror needs no
credentials; pulling ``dhi.io`` directly requires a ``docker login dhi.io``.
For the 3.4.x line Airflow also publishes ``legacy`` images, built the way all images were built before
3.4.0: Python is downloaded from python.org, its signature verified and compiled on a ``debian:bookworm-slim``
base. They are deprecated and will be removed in Airflow 3.5.0. To build one yourself, set these arguments:
.. code-block:: bash
docker build . --build-arg AIRFLOW_IMAGE_FLAVOR="legacy" --build-arg BASE_IMAGE="debian:bookworm-slim" \
--build-arg AIRFLOW_PYTHON_VERSION="3.13"
List of default extras in the production Dockerfile:
.. BEGINNING OF EXTRAS LIST UPDATED BY PRE COMMIT
* aiobotocore
* amazon
* async
* celery
* cncf-kubernetes
* common-io
* common-messaging
* docker
* elasticsearch
* fab
* ftp
* git
* google
* google-auth
* graphviz
* grpc
* hashicorp
* http
* ldap
* microsoft-azure
* mysql
* odbc
* openlineage
* opensearch
* pandas
* postgres
* redis
* sendgrid
* sftp
* slack
* snowflake
* ssh
* statsd
* uv
.. END OF EXTRAS LIST UPDATED BY PRE COMMIT
Image optimization options
..........................
The main advantage of Customization method of building Airflow image, is that it allows to build highly optimized image because
the final image (RUNTIME) might not contain all the dependencies that are needed to build and install all other dependencies
(DEV). Those arguments allow to control what is installed in the DEV image and what is installed in RUNTIME one, thus
allowing to produce much more optimized images. See :ref:`Building optimized images<image-build-optimized>`.
for examples of using those arguments.
+-------------------------------------+------------------------------------------+------------------------------------------+
| Build argument | Default value | Description |
+=====================================+==========================================+==========================================+
| ``UPGRADE_RANDOM_INDICATOR_STRING`` | | If set to a random, non-empty value |
| | | the dependencies are upgraded to newer |
| | | versions. In CI it is set to build id |
| | | to make sure subsequent builds are not |
| | | reusing cached images with same value. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_PYTHON_DEPS`` | | Optional python packages to extend |
| | | the image with some extra dependencies. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``DEV_APT_COMMAND`` | | Dev apt command executed before dev deps |
| | | are installed in the Build image. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_DEV_APT_COMMAND`` | | Additional Dev apt command executed |
| | | before dev dep are installed |
| | | in the Build image. Should start with |
| | | ``&&``. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``DEV_APT_DEPS`` | Empty - install default dependencies | Dev APT dependencies installed |
| | (see ``install_os_dependencies.sh``) | in the Build image. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_DEV_APT_DEPS`` | | Additional apt dev dependencies |
| | | installed in the Build image. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_DEV_APT_ENV`` | | Additional env variables defined |
| | | when installing dev deps. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``RUNTIME_APT_COMMAND`` | | Runtime apt command executed before deps |
| | | are installed in the ``main`` stage. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_RUNTIME_APT_COMMAND`` | | Additional Runtime apt command executed |
| | | before runtime dep are installed |
| | | in the ``main`` stage. Should start with |
| | | ``&&``. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``RUNTIME_APT_DEPS`` | Empty - install default dependencies | Runtime APT dependencies installed |
| | (see ``install_os_dependencies.sh``) | in the Main image. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_RUNTIME_APT_DEPS`` | | Additional apt runtime dependencies |
| | | installed in the Main image. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``ADDITIONAL_RUNTIME_APT_ENV`` | | Additional env variables defined |
| | | when installing runtime deps. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``INSTALL_MYSQL_CLIENT`` | ``true`` | Whether MySQL client should be installed |
| | | The mysql extra is removed from extras |
| | | if the client is not installed. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``INSTALL_MYSQL_CLIENT_TYPE`` | ``mariadb`` | Type of MySQL client library. This |
| | | can be ``mariadb`` or ``mysql`` |
| | | Regardless of the parameter, ``mariadb`` |
| | | will always be used on ARM. |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``INSTALL_MSSQL_CLIENT`` | ``true`` | Whether MsSQL client should be installed |
+-------------------------------------+------------------------------------------+------------------------------------------+
| ``INSTALL_POSTGRES_CLIENT`` | ``true`` | Whether Postgres client should be |
| | | installed |
+-------------------------------------+------------------------------------------+------------------------------------------+
Installing Airflow using different methods
..........................................
Those parameters are useful only if you want to install Airflow using different installation methods than the default
(installing from PyPI packages).
This is usually only useful if you have your own fork of Airflow and want to build the images locally from
those sources - either locally or from packages. This way you do not need to release your
Airflow and Providers via PyPI - they can be installed directly from sources.
Another option of installation is to build Airflow from previously prepared binary Python packages which might
be useful if you need to build Airflow in environments that require high levels of security.
You can see some examples of those in:
* :ref:`Using custom installation sources<image-build-custom>`,
* :ref:`Build images in security restricted environments<image-build-secure-environments>`
+------------------------------------------+------------------------------------------+------------------------------------------+
| Build argument | Default value | Description |
+==========================================+==========================================+==========================================+
| ``AIRFLOW_INSTALLATION_METHOD`` | ``apache-airflow`` | Installation method of Apache Airflow. |
| | | ``apache-airflow`` for installation from |
| | | PyPI or "." to install from |
| | | local sources. Installing from sources |
| | | requires appropriate values of the |
| | | ``AIRFLOW_SOURCES_FROM`` and |
| | | ``AIRFLOW_SOURCES_TO`` variables (see |
| | | below) |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_SOURCES_FROM`` | ``Dockerfile`` | Sources of Airflow. Set it to "." when |
| | | you install Airflow from local sources |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_SOURCES_TO`` | ``/Dockerfile`` | Target for Airflow sources. Set to |
| | | "/opt/airflow" when you install Airflow |
| | | from local sources. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_VERSION_SPECIFICATION`` | | Optional - might be used for using limit |
| | | for Airflow version installation - for |
| | | example ``<2.0.2`` for automated builds. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``AIRFLOW_CONSTRAINTS_LOCATION`` | | If not empty, it will override the |
| | | source of the constraints with the |
| | | specified URL or file. Note that the |
| | | file has to be in Docker context so |
| | | it's best to place such file in |
| | | one of the folders included in |
| | | ``.dockerignore`` file. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``DOCKER_CONTEXT_FILES`` | ``Dockerfile`` | If set to a folder (for example to |
| | | ``docker-context-files`` folder), then |
| | | this folder will be copied to the |
| | | ``docker-context-files`` inside the |
| | | context of docker and you will be able |
| | | to install from binary files present |
| | | there. By default we set it to |
| | | Dockerfile as we know the file is there, |
| | | otherwise the COPY instruction fails. |
+------------------------------------------+------------------------------------------+------------------------------------------+
| ``INSTALL_DISTRIBUTIONS_FROM_CONTEXT`` | ``false`` | If set to true, Airflow, providers and |
| | | all dependencies are installed from |
| | | from locally built/downloaded |
| | | .whl and .tar.gz files placed in the |
| | | ``docker-context-files``. |
+------------------------------------------+------------------------------------------+------------------------------------------+
Caching dependencies
....................
We are using ``--mount-type=cache`` volumes to speed up installation of dependencies for Airflow images. Combined with ``uv``
speed and extensive use of caching, as well as quick restoring of the cache in CI environment, this allows us to build images
quickly - for both CI and local development purposes. The cache can be easily invalidated by providing a new value of
``DEPENDENCY_CACHE_EPOCH`` build argument or changing it inside the Dockerfile.
+------------------------------------+------------------------------------------+------------------------------------------+
| Build argument | Default value | Description |
+====================================+==========================================+==========================================+
+------------------------------------+------------------------------------------+------------------------------------------+
| ``DEPENDENCY_CACHE_EPOCH`` | ``"0"`` | Allow to invalidate cache by passing a |
| | | new argument. |
+------------------------------------+------------------------------------------+------------------------------------------+
Reference in new issue View Git Blame Copy Permalink