mirror of
https://github.com/apache/airflow.git
synced 2026-11-04 07:12:48 +00:00
* Base CI and PROD images on hardened Python images Squashed from 19 commits (backup branch: backup/hardened-python-base-images-pre-rebase) before rebasing onto main. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Base images on trixie and keep a legacy bookworm PROD image flavor The 3.4.x line keeps publishing the previous images - Python compiled from sources on bookworm - as a deprecated legacy flavor, so the PROD image has to be buildable both ways while the CI image only needs the hardened trixie one. A weekly canary builds and checks the legacy images so they do not rot between releases. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Support the trixie hardened images' Debian-packaged Python and test legacy images like regular ones The Debian 13 hardened images install Python as Debian packages under /usr instead of /opt/python, so the image has to find the base Python's prefix and the system-Python guard must not mistake those packages for Debian's own Python. The legacy canary now runs the regular PROD image tests, not just a smoke check, so the legacy images are covered as well as the default ones. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Drop CI image packages that Debian trixie no longer has The CI image is trixie-only now: libgcc-11-dev was a bookworm-era pin (build-essential brings the matching libgcc-14-dev) and software-properties-common was removed in Debian 13; nothing uses its add-apt-repository. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Trust Microsoft's newer signing key for the trixie MSSQL repository Microsoft signs its Debian 13 package repository with its 2025 general signing key rather than the 2015 release key, so apt on trixie rejected the repository as unsigned. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Restore the PAM configs the trixie hardened images strip The Debian 13 hardened images ship libpam-runtime and passwd already installed but without their /etc/pam.d files, so installing them restored nothing and adding the CI image's airflow user failed with pam_start() errors. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Document the anonymous Debian package repository of the hardened images The trixie hardened images add Docker's own package repository, whose rebuilt packages take precedence over Debian's - users extending the image should know where their packages come from, and that no credentials are needed for it. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Keep a second hardened Python out of the trixie images On trixie, libenchant-2-dev's build dependencies need python3, which Docker's package repository satisfies with its own python-3.13 - a second interpreter in the 3.12 image that broke building C extensions. pyenchant only needs the runtime library, which it finds through the ldconfig cache, and the guard now fails early on any hardened Python other than the image's own. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Install the Python build dependencies for the legacy image flavor Dropping the packages that only existed to compile Python left the legacy flavor compiling a Python without sqlite3, lzma or readline - CPython builds those modules only when their headers are present, and the image could not even run airflow. The build now also fails when one of them is missing. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Refresh the shared library cache after apt runs in hardened images The hardened images ship libc-bin without its dpkg trigger, so libraries installed by apt stayed invisible to ctypes.util.find_library() - the CI image's docs build failed with "The 'enchant' C library was not found". A dpkg post-invoke hook runs ldconfig after every apt run, also in images extending ours. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Warn about apt pulling a second Python into hardened images Extending a hardened image with apt packages that depend on python3 silently installs another Python next to the image's own, and C extensions then build against the wrong one - users need to know to check before adding such packages. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Add trixie and sigstore to the docs spelling wordlist Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Count only host keys when waiting for sshd in the CI image OpenSSH 10 in trixie prints a '# host:port' banner comment per connection to ssh-keyscan's stdout, so the line count never matched the three host keys and every test job waited until it timed out. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Invalidate the CI image mount cache for the trixie base The uv cache restored into CI image builds held wheels compiled against bookworm's libraries - an xmlsec wheel built against bookworm's MD5-enabled libxmlsec1 failed to import on trixie, which dropped MD5. The cache key does not encode the base OS, so it has to be bumped when that changes. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Install Java that exists on trixie in the Java examples Debian trixie ships no Java 17. The Java SDK e2e image keeps Java 17 for Spark 3.5 by taking it from the Temurin repository, and the docs example uses trixie's Java 21. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Let old C extensions build in the trixie CI image GCC 14 turns incompatible pointer types, integer conversions and implicit declarations into errors, and the hardened Python passes -Werror=implicit-function-declaration on to extensions - the lowest-dependency tests' old sdists (xmlsec 1.3.14, ibm-db 3.2.0) only built because those were warnings. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Document the stricter C extension builds of the hardened images Users extending the image with old source distributions hit the same GCC 14 errors as CI, and need to know how to relax them. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Export KinD logs when the lang-SDK K8S test setup fails A failing lang-SDK helm upgrade raised past the log export that every other step of the complete K8S test run does, so the cluster was deleted with nothing saved and the job's KinD logs artifact stayed empty. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x * Keep apt from replacing the hardened images' Python Debian 13's python3 packages conflict with the hardened images' python-3.X packages, so installing NodeSource's nodejs (which depends on python3) into the lang-SDK runtime image made apt remove the image's Python 3.11, and the migration pods crash-looped. The image now holds its Python packages, so such an install fails instead, and the runtime image copies the node binary from the official node image rather than installing the deb. Generated-by: Claude Opus 5 * Keep perl-base in step with the perl apt can install The hardened apt repository publishes git, which depends on perl, but not perl itself, while the hardened images ship their own build of perl-base. Debian's perl depends on exactly its own perl-base version, so apt refused to install git and the hardened CI image build failed. The image now moves perl-base to the version of the installable perl before installing its packages; when the hardened repository publishes a matching perl, the versions agree and nothing changes. Generated-by: Claude Opus 5 * Mirror the hardened base images only from main The mirror is the default BASE_IMAGE of every Airflow build, and the job holds packages:write plus the Docker Hub credentials. Scheduled runs are always on main, but anyone with write access could dispatch the workflow from another branch and replace the mirrored images. Generated-by: Claude Opus 5 * Check the Adoptium signing key fingerprint in the Java e2e image The key is downloaded over HTTPS at build time; pinning its fingerprint means a replaced key on the server fails the build instead of being trusted for every package from that repository. Generated-by: Claude Opus 5 * Allow mirroring the hardened base images from v3-*-test branches The release branches build images from the same mirror, so maintainers need to refresh it from there too. Generated-by: Claude Opus 5
470 lines
31 KiB
ReStructuredText
470 lines
31 KiB
ReStructuredText
.. Licensed to the Apache Software Foundation (ASF) under one
|
|
or more contributor license agreements. See the NOTICE file
|
|
distributed with this work for additional information
|
|
regarding copyright ownership. The ASF licenses this file
|
|
to you under the Apache License, Version 2.0 (the
|
|
"License"); you may not use this file except in compliance
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
.. Unless required by applicable law or agreed to in writing,
|
|
software distributed under the License is distributed on an
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations
|
|
under the License.
|
|
|
|
Dockerfile Changelog
|
|
====================
|
|
|
|
The ``Dockerfile`` does not strictly follow the `SemVer <https://semver.org/>`_ approach of
|
|
Apache Airflow when it comes to features and backwards compatibility. While Airflow code strictly
|
|
follows it, the ``Dockerfile`` is really a way to give users a conveniently packaged Airflow
|
|
using standard container approach, so occasionally there are some changes in the building process
|
|
or in the entrypoint of the image that require slight adaptation of how it is used or built.
|
|
|
|
The Changelog below describes the changes introduced in each version of the docker images released by
|
|
the Airflow team.
|
|
|
|
:note: The Changelog below concerns only the convenience production images released at
|
|
`Airflow DockerHub <https://hub.docker.com/r/apache/airflow>`_ . The images that are released
|
|
there are usually built using the ``Dockerfile`` released together with Airflow. However, you are
|
|
free to take latest released ``Dockerfile`` from Airflow and use it to build an image for
|
|
any Airflow version from the ``Airflow 2`` line. There is no guarantee that it will work, but if it does,
|
|
then you can use latest features from that image to build images for previous Airflow versions.
|
|
|
|
Airflow 3.4.0
|
|
~~~~~~~~~~~~~
|
|
|
|
* The image does not support Python 3.10 anymore as Python 3.10 reaches end of life in October 2026.
|
|
Images are released for Python 3.11, 3.12, 3.13 and 3.14.
|
|
|
|
* The ``tdsodbc`` package was added to the image so that the FreeTDS ODBC driver
|
|
(``libtdsodbc.so``) is available for connecting to Sybase/TDS databases via ODBC.
|
|
|
|
* The ``libxmlsec1-openssl`` package was added to the image. ``libxmlsec1`` contains no crypto
|
|
engine of its own, so ``import xmlsec`` (used by ``python3-saml``) failed with
|
|
``libxmlsec1-openssl.so.1: cannot open shared object file``.
|
|
|
|
In Airflow 3.4.0 the base image changed again - from ``debian:bookworm-slim`` with a Python compiled in
|
|
the image, to the Debian 13 "trixie" `Docker Hardened Image <https://dhi.io>`_ for Python, which already
|
|
carries a Python built by Docker. This removes the Python compilation, and the source download and signature verification
|
|
that went with it, from the build entirely - the OS dependency layer of the production image went from
|
|
roughly 210s to 105s in a local cache-disabled build.
|
|
|
|
* The ``BASE_IMAGE`` arg now defaults to ``ghcr.io/apache/airflow/base/python:<version>-debian13-dev``,
|
|
which is Airflow's public mirror of the upstream ``dhi.io/python`` image. Pulling from ``dhi.io``
|
|
directly requires a ``docker login dhi.io``; pulling the mirror requires nothing. If you pass your own
|
|
``BASE_IMAGE``, it now has to be an image that already provides Python - a bare ``debian:bookworm-slim``
|
|
only works with ``AIRFLOW_IMAGE_FLAVOR="legacy"``.
|
|
* The image is based on Debian 13 "trixie" instead of Debian 12 "bookworm". Library packages renamed in
|
|
trixie's 64-bit ``time_t`` transition (for example ``libssl3t64``, ``libldap2``) have to be used in
|
|
custom ``RUNTIME_APT_DEPS``, and the MySQL-compatible client comes from MariaDB 11.8 (with
|
|
``mariadb-client-compat`` providing the ``mysql`` named commands).
|
|
* The new ``AIRFLOW_IMAGE_FLAVOR`` arg (``hardened`` by default) can be set to ``legacy`` to build the
|
|
image as before: Python compiled from sources on a plain Debian ``BASE_IMAGE`` such as
|
|
``debian:bookworm-slim``. Images of that flavor are published for the 3.4.x line, are deprecated, and
|
|
will be removed in Airflow 3.5.0. The flavor is recorded in the ``org.apache.airflow.image.flavor`` label.
|
|
* The ``PYTHON_LTO`` arg was removed. It only controlled Link-Time Optimization while compiling Python;
|
|
the legacy flavor always compiles with LTO. To build a FIPS-compliant image, point ``BASE_IMAGE`` at a FIPS
|
|
variant of the hardened image, for example ``dhi.io/python:3.13.16-debian13-fips-dev`` (those variants
|
|
require a paid Docker subscription).
|
|
* Python is installed under ``/usr`` by the base image, as Debian packages. ``/usr/python`` is a symlink to it, and the
|
|
``/usr/local/bin`` symlinks are unchanged, so paths that worked before keep working.
|
|
* The hardened images ship the standard library with no ``.pyc`` files, so the build compiles it into the
|
|
image. This keeps the behaviour introduced in Airflow 3.1.4 (see below): the standard library is owned
|
|
by root while the image runs as ``airflow``, so a missing bytecode cache could never be filled and every
|
|
import would leak a negative ``dentry``.
|
|
* Python in the hardened images is not the plain upstream build: Docker patches it (``pkgutil.get_data()``
|
|
rejects ``..`` resource paths), builds it without profile-guided optimization, and compiles it with a
|
|
1 MiB default thread stack. The image restores the previous 8 MiB thread stack. See
|
|
:ref:`image-hardened-base` for these and the other differences from the previous images.
|
|
|
|
As with any base image change, some ``apt`` packages that used to be present as a side effect are not
|
|
there any more. The hardened images are deliberately minimal - they ship no compiler, no ``curl``,
|
|
``wget``, ``git`` or ``gzip``, and a stripped ``/etc`` - so a custom image that relied on something being
|
|
present may need to install it explicitly. See :doc:`Building the image <build>`.
|
|
|
|
Airflow 3.1.4
|
|
~~~~~~~~~~~~~
|
|
|
|
In Airflow 3.1.4, the images are build without removing of .pyc and .pyo files when Python is built.
|
|
This increases the size of the image slightly (<0.5%), but improves performance of Python in the container
|
|
because Python does not need to recompile the files on the first run but more importantly, if you use
|
|
``exec`` to run Health Checks, removed .pyc files caused a small but ever growing memory leak in the Unix
|
|
kernel connected to negative ``dentries`` created when .pyc files were attempted to be compiled and failed.
|
|
This over time could lead to out-of-memory issues on the host running the container.
|
|
|
|
More information about ``dentries`` can be found in `this article <https://lwn.net/Articles/814535/>`_.
|
|
|
|
Airflow 3.1.0
|
|
~~~~~~~~~~~~~
|
|
|
|
In Airflow 3.1.0 we changed the base images used for building the Airflow images. Previously the images
|
|
were based on "official" Python images from DockerHub, however those images sometimes lag behind the
|
|
latest security patches and their maintainers made decisions about using older setuptools and pip versions,
|
|
however we want to be able to use the latest versions of those tools to build the images. Therefore
|
|
we decided to use ``bookworm-slim`` images from Debian as the base images for Airflow images and we compile
|
|
and install the latest Python version in the image based on the official packages from the Python Software
|
|
Foundation. This change should be transparent for those who extend the images or use them "as is", however
|
|
for those who want to build custom images, there are changes to the build-args passed to the ``docker build``
|
|
command:
|
|
|
|
* The ``PYTHON_BASE_IMAGE`` arg is no longer supported - the base image is always ``debian:bookworm-slim``
|
|
* Instead ``AIRFLOW_PYTHON_VERSION`` arg is introduced to select the Python version to be installed in the
|
|
image. It should be full Python version - for example ``3.12.7`` or two-digit version like ``3.12``
|
|
(the latest released Python patch version will be installed in that case).
|
|
* Python is now installed in ``/usr/python/`` directory not in ``/usr/local/`` as it was previously.
|
|
This should not affect regular uses because there are symlinks from ``/usr/local/bin/`` to
|
|
``/usr/python/bin/``, and ``/usr/python/bin`` is first on the PATH, however if the ``/usr/local/``
|
|
location has been hard-coded to - for example - detect ``site-packages`` dir, the folders moved to
|
|
the ``/usr/python`` directory.
|
|
|
|
There might be other subtle changes in the image due to the change of the base image, some packages (
|
|
especially dev libraries installed with ``apt`` might not be installed by default if they were installed
|
|
as a side-effect in the original image, however that should only affect those who want to customise the image.
|
|
They should be able to install in their custom images following the :doc:`Building the image <build>`
|
|
|
|
The Python 3.13 image for Airflow 3.1.0 (both slim and regular) does not contain ``fab`` provider because
|
|
at the time of release the ``fab`` provider did not support Python 3.13. It should be possible to install
|
|
future versions of ``fab`` provider (when they support Python 3.13) in the image using ``pip install``,
|
|
and until it is possible - if you need ``fab`` provider (and particularly FABAuthManager) in the
|
|
image, you should use Python 3.12 image. You can use experimental KeycloakAuthManager in Python 3.13 image
|
|
or develop your own AuthManager.
|
|
|
|
Airflow 3.0.2
|
|
~~~~~~~~~~~~~
|
|
|
|
* The ``git`` binary was added to the image by default which is needed for the git provider to work.
|
|
|
|
Airflow 3.0.1
|
|
~~~~~~~~~~~~~
|
|
|
|
* The ``ARM`` image is not experimental any more - we are running the ARM tests regularly in our CI. The
|
|
``MySQL`` support for `ARM` images however is still experimental.
|
|
|
|
Airflow 3.0
|
|
~~~~~~~~~~~
|
|
|
|
* The ``virtualenv`` package is no longer installed in the reference container. Airflow 3 and standard
|
|
provider relies on ``venv`` module from Python standard library.
|
|
* There is no ``pipx`` and ``mssql-cli`` installed in the image by default. We recommend to use
|
|
``uv tool`` instead of ``pipx`` and ``mssql-cli`` is not used in the image by default as we do not
|
|
have mssql metadata support any more.
|
|
* The ``INSTALL_PACKAGES_FROM_CONTEXT`` arg changed to ``INSTALL_DISTRIBUTIONS_FROM_CONTEXT``
|
|
* The parameter ``UPGRADE_INVALIDATION_STRING`` is renamed to ``UPGRADE_RANDOM_INDICATOR_STRING``
|
|
|
|
Airflow 2.11.1
|
|
~~~~~~~~~~~~~~
|
|
|
|
* The image does not support Python 3.9 anymore as Python 3.9 reached end of life on October 31, 2025 and
|
|
The Airflow 2.11.1 has been released on February 20, 2026 and Airflow 2.11.1 does not support Python 3.9.
|
|
|
|
|
|
Airflow 2.10
|
|
~~~~~~~~~~~~
|
|
* The image does not support Debian-Bullseye(11) anymore. The image is based on Debian-Bookworm (12).
|
|
|
|
Airflow 2.9
|
|
~~~~~~~~~~~
|
|
|
|
* The "latest" image (i.e. default Airflow image when ``apache/airflow`` is used or
|
|
``apache/airflow:slim-latest``) uses now the newest supported Python version. Previously it was using
|
|
the "default" Python version which was Python 3.8 as of Airflow 2.8. With Airflow reference images
|
|
released for Airflow 2.9.0, the images are going to use Python 3.12 as this is the latest supported
|
|
version for Airflow 2.9 line. Users can use Python 3.8 by using ``apache/airflow:2.9.0-python3.8`` and
|
|
``apache/airflow:slim-2.9.0-python-3.8`` images respectively so while the change is potentially
|
|
breaking, it is very easy to switch to the previous behaviour.
|
|
|
|
* The ``PIP_USER`` flag is removed and replaced by ``VIRTUAL_ENV`` pointing to ``~/.local`` where Airflow
|
|
is installed. This has the effect that the Airflow installation is treated as a regular virtual environment,
|
|
but unlike a regular virtualenv, the ``~/.local`` directory is seen as ``system level`` and when the
|
|
worker creates dynamically the virtualenv with ``--system-site-packages`` flag, the Airflow installation and all
|
|
packages there are also present in the new virtualenv. When you do not use the flag, they are not
|
|
copied there which is a backwards-compatible behaviour with having ``PIP_USER`` set.
|
|
|
|
* The image contains latest ``uv`` binary (latest at the moment of release) - which is a new faster
|
|
replacement for ``pip``. While the image is still using ``pip`` by default, you can use ``uv``
|
|
to install packages and - experimentally - you can also build custom images with
|
|
``--arg AIRFLOW_USE_UV=true`` which will us ``uv`` to perform the installation. This is an experimental
|
|
support, as ``uv`` is very fast but also a very new feature in the Python ecosystem.
|
|
|
|
* Constraints used to install the image are available in "${HOME}/constraints.txt" now - you can use them
|
|
to install additional packages in the image without having to find out which constraints you should use.
|
|
|
|
* The image adds ``libev`` library to the image as it is required by cassandra driver for Python 3.12, also
|
|
``libev`` will be used in other Python versions as a more robust and faster way for cassandra driver
|
|
to handle events.
|
|
|
|
Airflow 2.8
|
|
~~~~~~~~~~~
|
|
* 2.8.3
|
|
|
|
* The ``gosu`` binary was removed from the image. This is a potentially breaking change for users who relied on
|
|
``gosu`` to change the user in the container. The ``gosu`` binary was removed because it was a source of
|
|
security vulnerabilities as it was linked against older Go standard libraries.
|
|
|
|
* The ``smtp`` provider is now included in the list of providers installed by default in the image.
|
|
|
|
* 2.8.1
|
|
|
|
* Fixed a discrepancy in MySQL client libraries. In 2.8.0 if not specify ``INSTALL_MYSQL_CLIENT_TYPE`` build arg
|
|
during build custom X86 image by default packages would be compiled by using **MariaDB** libraries,
|
|
however **MySQL** libraries were installed in the final image.
|
|
|
|
* 2.8.0
|
|
|
|
* Add ``libxmlsec1`` and ``libxmlsec1-dev`` libraries to dev PROD image and ``libxmlsec1`` library to runtime PROD
|
|
image as it is required by ``python3-saml`` library.
|
|
|
|
* The image is based on ``Debian Bookworm`` in 2.8.0 rather than ``Debian Bullseye``. This might cause some
|
|
problems when building custom images. You are advised to make sure your system level dependencies are
|
|
working with ``Debian Bookworm``. While all reference images of Airflow 2.8.0 are built on ``Debian Bookworm``,
|
|
it is still possible to build deprecated custom ``Debian Bullseye`` based image in 2.8.0 following the
|
|
|
|
* By default the images now have "MariaDB" client installed. Previous images had "MySQL" client installed.
|
|
The MariaDB client is a drop-in replacement for "MySQL" one and is compatible with MySQL. This might
|
|
be a breaking change for users who used MySQL client in their images, however those should be very
|
|
specific cases and vast majority of users should not see any difference. Users can still use
|
|
MySQL client by setting ``INSTALL_MYSQL_CLIENT_TYPE=mysql`` build arg and build the custom X86 image.
|
|
The ARM image always uses MariaDB client, this argument is ignored. The "mysql" apt repository is
|
|
removed from the /etc/apt/sources.list.d/ and if you want to install anything from this repository when
|
|
extending the images, you need to manually add the right key and repository in your Dockerfile,
|
|
following the instructions in `A Quick Guide to Using the MySQL APT repository <https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/>`_.
|
|
|
|
Airflow 2.7
|
|
~~~~~~~~~~~
|
|
|
|
* 2.7.3
|
|
|
|
* Add experimental feature for select type of MySQL Client libraries during the build custom image via ``INSTALL_MYSQL_CLIENT_TYPE``
|
|
build arg. ``mysql`` for install MySQL client libraries from `Oracle APT repository <https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/>`_,
|
|
``mariadb`` for install MariaDB client libraries from `MariaDB repository <https://mariadb.com/kb/en/mariadb-package-repository-setup-and-usage/#mariadb-repository>`_.
|
|
The selection of MySQL Client libraries only available on AMD64 (x86_64) for ARM docker image it will always install
|
|
MariaDB client.
|
|
|
|
* Docker CLI version in the image is bumped to 24.0.6 version.
|
|
|
|
* PIP caching for local builds has been enabled to speed up local custom image building
|
|
|
|
* 2.7.0
|
|
|
|
* As of now, Python 3.7 is no longer supported by the Python community. Therefore, to use Airflow 2.7.0, you must ensure your Python version is
|
|
either 3.8, 3.9, 3.10, or 3.11.
|
|
|
|
Airflow 2.6
|
|
~~~~~~~~~~~~~
|
|
|
|
* 2.6.3
|
|
|
|
* Add ``libgeos-dev`` library to runtime PROD image as it is required by BigQuery library on ARM image
|
|
|
|
|
|
* 2.6.0
|
|
|
|
* Snowflake provider installed by default
|
|
|
|
* The ARM experimental image adds support for MySQL via MariaDB client libraries.
|
|
|
|
Airflow 2.5
|
|
~~~~~~~~~~~
|
|
|
|
* 2.5.1
|
|
|
|
* The ARM experimental image adds support for MSSQL
|
|
|
|
* 2.5.0
|
|
|
|
* The docker CLI binary is now added to the images by default (available on PATH). Version 20.10.9 is used.
|
|
|
|
Airflow 2.4
|
|
~~~~~~~~~~~
|
|
|
|
* 2.4.0
|
|
|
|
* You can specify additional ``pip install`` flags when you build the image via ``ADDITIONAL_PIP_INSTALL_FLAGS``
|
|
build arg.
|
|
* Support for ``Debian Buster`` was dropped, including the possibility of building customized images as
|
|
``Debian Buster`` reached end of life.
|
|
|
|
Airflow 2.3
|
|
~~~~~~~~~~~
|
|
|
|
* 2.3.0
|
|
|
|
* Airflow 2.3 ``Dockerfile`` is now better optimized for caching and "standalone" which means that you
|
|
can copy **just** the ``Dockerfile`` to any folder and start building custom images. This,
|
|
however, requires `Buildkit <https://docs.docker.com/develop/develop-images/build_enhancements/>`_
|
|
to build the image because we started using features that are only available in ``Buildkit``.
|
|
This can be done by setting ``DOCKER_BUILDKIT=1`` as an environment variable
|
|
or by installing `the buildx plugin <https://docs.docker.com/buildx/working-with-buildx/>`_
|
|
and running ``docker buildx build`` command.
|
|
* MySQL is experimentally supported on ARM through MariaDB client library
|
|
* Add Python 3.10 support
|
|
* Switch to ``Debian Bullseye`` based images. ``Debian Buster`` is deprecated and only available for
|
|
customized image building.
|
|
* Add Multi-Platform support (AMD64/ARM64) in order to accommodate MacOS M1 users
|
|
* Build parameters which control if packages and Airflow should be installed from context file were
|
|
unified
|
|
* The ``INSTALL_FROM_PYPI`` arg was removed - it is automatically detected now.
|
|
* The ``INSTALL_FROM_DOCKER_CONTEXT_FILES`` arg changed to ``INSTALL_PACKAGES_FROM_CONTEXT``
|
|
|
|
Airflow 2.2
|
|
~~~~~~~~~~~
|
|
|
|
* 2.2.4
|
|
* Add support for both ``.piprc`` and ``pip.conf`` customizations
|
|
* Add ArtifactHub labels for better discovery of the images
|
|
* Update default Python image to be 3.7
|
|
* Build images with ``Buildkit`` (optional)
|
|
* Fix building the image on Azure with ``text file busy`` error
|
|
|
|
* 2.2.3
|
|
* No changes
|
|
|
|
* 2.2.2
|
|
* No changes
|
|
|
|
* 2.2.1
|
|
* Workaround the problem with ``libstdcpp`` TLS error
|
|
|
|
* 2.2.0
|
|
* Remove AIRFLOW_GID (5000) from Airflow images (potentially breaking change for users using it)
|
|
* Added warnings for Quick-start docker compose
|
|
* Fix warm shutdown for celery worker (signal propagation)
|
|
* Add Oauth libraries to PROD images
|
|
* Add Python 3.9 support
|
|
|
|
Airflow 2.1
|
|
~~~~~~~~~~~
|
|
|
|
* MySQL changed the keys to sign their packages on 17 Feb 2022. This caused all released images
|
|
to fail when being extended. As result, on 18 Feb 2021 we re-released all
|
|
the ``2.2`` and ``2.1`` images with latest versions of ``Dockerfile``
|
|
containing the new signing key.
|
|
|
|
There were subtle changes in the behaviour of some 2.1 images due to that (more details below)
|
|
Detailed `issue here <https://github.com/apache/airflow/issues/20911>`_
|
|
|
|
:note: that the changes below were valid before image refreshing on 18 Feb 2022.
|
|
Since all the images were refreshed on 18 Feb with the same ``Dockerfile``
|
|
as 2.1.4, the changes 2.1.1 -> 2.1.3 are
|
|
effectively applied to all the images in 2.1.* line.
|
|
The images refreshed have also those fixes added:
|
|
|
|
* All 2.1.* image versions refreshed on 18 Feb 2022 have those fixes applied:
|
|
* Fix building the image on Azure with ``text file busy`` error
|
|
* Workaround the problem with ``libstdcpp`` TLS error
|
|
* Remove AIRFLOW_GID (5000) from Airflow images (potentially breaking change for users using it)
|
|
* Added warnings for Quick-start docker compose
|
|
* Add Oauth libraries to PROD images
|
|
|
|
Original image Changelog (before the refresh on 18 Feb 2022):
|
|
|
|
* 2.1.4
|
|
* Workaround the problem with ``libstdcpp`` TLS error
|
|
* fixed detection of port number in connection URL
|
|
* Improve warnings for quick-start-docker compose
|
|
* Fix warm shutdown for celery worker (signal propagation)
|
|
|
|
* 2.1.3
|
|
* fixed auto-creation of user to use non-deprecated ``create user`` rather than ``user_create``
|
|
* remove waiting for celery backend for ``worker`` and ``flower`` commands rather than ``scheduler`` and ``celery`` only
|
|
* remove deprecated ``airflow upgradedb`` command from Airflow 1.10 in case upgrade is requested
|
|
* Add Python 3.9 support
|
|
|
|
* 2.1.2
|
|
* No changes
|
|
|
|
* 2.1.1
|
|
* Fix failure of lack of default commands (failed when no commands were passed)
|
|
* Added ``_PIP_ADDITIONAL_REQUIREMENTS`` development feature
|
|
|
|
* 2.1.0
|
|
* Unset default ``PIP_USER`` variable - which caused PythonVirtualEnv to fail
|
|
|
|
Airflow 2.0
|
|
~~~~~~~~~~~
|
|
|
|
* MySQL changed the keys to sign their packages on 17 Feb 2022. This caused all released images
|
|
to fail when being extended. As result, on 18 Feb 2021 we re-released all
|
|
the ``2.2`` and ``2.1`` images with latest versions of ``Dockerfile``
|
|
containing the new signing key.
|
|
|
|
There were no changes in the behaviour of 2.0.2 image due to that
|
|
Detailed `issue here <https://github.com/apache/airflow/issues/20911>`_ .
|
|
Only 2.0.2 image was regenerated, as 2.0.1 and 2.0.0 versions are hardly used and it is unlikely someone
|
|
would like to extend those images. Extending 2.0.1 and 2.0.0 images will lead to failures of "missing key".
|
|
|
|
* 2.0.2
|
|
* Set correct PYTHONPATH for ``root`` user. Allows to run the image as root
|
|
* Warn if the deprecated 5000 group ID was used for airflow user when running the image
|
|
(should be 0 for the OpenShift compatibility). Fails if the group 5000 was used with any other user
|
|
(it would not work anyway but with cryptic errors)
|
|
* Set umask as 002 by default, so that you can actually change the user id used to run the image
|
|
(required for OpenShift compatibility)
|
|
* Skip checking the DB and celery backend if CONNECTION_CHECK_MAX_COUNT is equal to 0
|
|
|
|
* 2.0.1
|
|
* Avoid reverse IP lookup when checking DB availability. This helped to solve long delays on misconfigured
|
|
docker engines
|
|
* Add auto-detection of redis and amqp broker ports
|
|
* Fixed detection of all user/password combinations in URLs - helps in auto-detecting ports and testing
|
|
connectivity
|
|
* Add possibility to create Admin user automatically when entering the image
|
|
* Automatically create system user when different user than ``airflow`` is used. Needed for OpenShift
|
|
compatibility
|
|
* Allows to exec to ``bash`` or ``python`` if specified as parameters
|
|
* Remove ``airflow`` command if it is specified as first parameter of the ``run`` command
|
|
|
|
* 2.0.0
|
|
* Initial release of the image based on Debian Buster
|
|
|
|
|
|
Changes after publishing the images
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Occasionally our images need to be regenerated using newer ``Dockerfiles`` or constraints.
|
|
This happens when an issue is found or a breaking change is released by our dependencies
|
|
that invalidates the already released image, and regenerating the image makes it usable again.
|
|
While we cannot assure 100% backwards compatibility when it happens, we at least document it
|
|
here so that users affected can find the reason for the changes.
|
|
|
|
|
|
+--------------+-------------+-----------------------------------------------------------------------------------+-------------------------------------------------------+------------------------------------------------+
|
|
| Date | Versions | Potentially breaking change | Reason | Link to Pull Request / Issue |
|
|
+==============+=============+===================================================================================+=======================================================+================================================+
|
|
| 19 July 2026 | 3.3.0 | * The ``apache-airflow-providers-databricks`` provider was upgraded from 6.13.0 | ``thrift`` 0.23.0 conflicted with | https://github.com/apache/airflow/issues/69603 |
|
|
| | | to 7.16.1 in the Python 3.11 and 3.12 images. | ``databricks-sql-connector`` 4.x (which requires | |
|
|
| | | * ``databricks-sql-connector`` upgraded from 2.0.2 to 4.2.5 and ``thrift`` from | ``thrift<0.21.0``), so the resolver kept the newer | |
|
|
| | | 0.23.0 to 0.16.0. | thrift and downgraded the connector, dragging the | |
|
|
| | | | databricks provider back to 6.13.0 (Oct 2024) with | |
|
|
| | | | import errors on 3.3.0. | |
|
|
+--------------+-------------+-----------------------------------------------------------------------------------+-------------------------------------------------------+------------------------------------------------+
|
|
| 28 Feb 2026 | 2.11.1 | * The grpcio==1.78.0 is used instead of 1.78.1 in Python 3.12 images. | The 1.78.1 version was yanked - caused major outage. | https://github.com/grpc/grpc/issues/41725. |
|
|
+--------------+-------------+-----------------------------------------------------------------------------------+-------------------------------------------------------+------------------------------------------------+
|
|
| 23 Feb 2026 | 2.11.1 | * Several dependencies upgraded with minor versions/patchlevels | Virtualenv project pulling out released 20.37.0 | https://github.com/pypa/virtualenv/issues/3061 |
|
|
| | | * See https://github.com/apache/airflow/commit/4a55543 for details | | |
|
|
+--------------+-------------+-----------------------------------------------------------------------------------+-------------------------------------------------------+------------------------------------------------+
|
|
| 19 July 2025 | 3.0.3 | * The ``standard`` provider upgraded from 1.4.0 to 1.4.1 | Sensor skipping issue | https://github.com/apache/airflow/pull/53455 |
|
|
+--------------+-------------+-----------------------------------------------------------------------------------+-------------------------------------------------------+------------------------------------------------+
|
|
| 24 Jun 2025 | 3.0.2 | * The ``fab`` provider upgraded from 2.2.0 to 2.2.1 | FAB provider user creation did not work | https://github.com/apache/airflow/issues/51854 |
|
|
| | | * ``common.messaging`` provider upgraded from 1.0.2 to 1.0.3 | importing SQS message failed with circular import | https://github.com/apache/airflow/issues/51770 |
|
|
| | | * git binary is added to the image | git bundle need it | https://github.com/apache/airflow/pull/51580 |
|
|
+--------------+-------------+-----------------------------------------------------------------------------------+-------------------------------------------------------+------------------------------------------------+
|
|
| 02 Aug 2024 | 2.9.3 | * The ``apache-airflow-providers-fab`` upgraded from 1.2.1 to 1.2.2 | FAB provider logout did not work for 2.9.3 | https://github.com/apache/airflow/issues/40922 |
|
|
+--------------+-------------+-----------------------------------------------------------------------------------+-------------------------------------------------------+------------------------------------------------+
|
|
| 12 Mar 2024 | 2.8.3 | * The image was refreshed with new dependencies (pandas < 2.2 and SMTP provider | Both dependencies caused breaking changes | https://github.com/apache/airflow/pull/37748 |
|
|
| | | 1.6.1) | | https://github.com/apache/airflow/pull/37701 |
|
|
+--------------+-------------+-----------------------------------------------------------------------------------+-------------------------------------------------------+------------------------------------------------+
|
|
| 16 Dec 2023 | All 2.* | * The AIRFLOW_GID 500 was removed | MySQL repository is removed after the key expiry | https://github.com/apache/airflow/issues/36231 |
|
|
| | | * MySQL ``apt`` repository key changed. | fiasco | |
|
|
+--------------+-------------+-----------------------------------------------------------------------------------+-------------------------------------------------------+------------------------------------------------+
|
|
| 17 June 2022 | 2.2.5 | * The ``Authlib`` library downgraded from 1.0.1 to 0.15.5 version | Flask App Builder not compatible with Authlib >= 1.0 | https://github.com/apache/airflow/pull/24516 |
|
|
| | 2.3.0-2.3.2 | | | |
|
|
+--------------+-------------+-----------------------------------------------------------------------------------+-------------------------------------------------------+------------------------------------------------+
|
|
| 18 Jan 2022 | All 2.2.*, | * The AIRFLOW_GID 500 was removed | MySQL changed keys to sign their packages on 17 Jan | https://github.com/apache/airflow/pull/20912 |
|
|
| | 2.1.* | * MySQL ``apt`` repository key changed. | 2022 | |
|
|
+--------------+-------------+-----------------------------------------------------------------------------------+-------------------------------------------------------+------------------------------------------------+
|