Commit Graph
26 Commits
Author SHA1 Message Date
Rahul Vats 47c02d3601 Upgrade the Airflow version to 3.5.0 and Task SDK version to 1.5.0 (#76388) 2026-11-04 00:32:18 +00:00
Jarek PotiukandRahul Vats 48435351c4 Pin the hardened base images to digests the CI has passed on (#76023)
* Refresh the hardened base image mirror every six hours

Docker rebuilds the hardened images in place and moves the dhi.io apt
repository on at the same time. With a weekly mirror, the base images
Airflow builds on lag the repository by up to a week, and any apt
install whose dependencies were rebuilt in between fails: build-essential
currently cannot be installed because the mirrored base has an older
libbz2 than the bzip2 the repository offers
(docker-hardened-images/catalog#689). Mirroring copies manifests between
registries without pulling layers, so running it several times a day is
cheap.

Generated-by: Claude Opus 5

* Pin the hardened base images to digests the CI has passed on

Building on the mirror's moving tags made CI non-reproducible: any
refresh of the mirror changed the image every build started from,
without a change in the repository and without the full CI having run
on it. A CI run could pass or fail depending only on when it ran, and
rebuilding the same commit later gave a different image.

The images now build on a digest, and only the upgrade check moves it,
in a PR that must pass the full CI before the new base is used. The
mirror refreshes every six hours and triggers that upgrade check when it
moves a tag past its pin, so the pins follow Docker's rebuilds - and the
dhi.io apt repository moving with them - within hours rather than drift
behind it. Every pinned and mirrored digest keeps a tag of its own, so a
pinned image never becomes untagged.

The hardened images keep stock Debian's apt sources beside dhi.io, and
apt took whichever offered the higher version, mixing Debian builds into
an image whose dependencies were built by Docker. dhi.io is now
preferred whenever it publishes the package.

Generated-by: Claude Opus 5

* Let dhi.io's package versions win over the image's own

Docker renumbered its builds ("1.0.8-6+dhi2" became
"1.0.8-6+deb13u0+dhi2", which sorts lower) and its apt repository moved
to the new numbers before its images did. Every package depending on a
renumbered build - bzip2, libffi-dev, libgpg-error-dev among them - is
then uninstallable, because apt will not replace the image's
higher-numbered copy with the repository's lower-numbered one, and the
CI image build fails (docker-hardened-images/catalog#689). Every package
in the image comes from dhi.io, so letting its repository decide their
versions is safe, and it keeps the build working through such a
renumbering.

Generated-by: Claude Opus 5

* Move the base images to Docker's current Python patch releases

Docker stops rebuilding a Python patch release once it publishes the
next, so the 3.11.17, 3.12.15 and 3.13.16 images still carry the old
package numbering that the dhi.io apt repository has moved past, and
installing packages on them fails (docker-hardened-images/catalog#689).
Docker fixed it in the images of the current patch releases.

Generated with the Python part of the upgrade check, which also pins
the new images to the digests the mirror serves for them.

Generated-by: Claude Opus 5

* Compare the registry host exactly in the mirror test

Generated-by: Claude Opus 5

* Lower the dhi.io apt pin below 1000 so dependencies can resolve

At priority 1001 the Package:* dhi.io preference forces dhi.io's build even as a downgrade, so apt keeps dhi.io's libpq5 17.11 and postgresql-client-18 (which needs libpq5 >= 18.6 from PGDG) cannot be installed. Below 1000 the preference still favours dhi.io for the packages it publishes but lets apt take a newer version from another repository to satisfy a dependency. The forced "downgrade" the >1000 priority handled no longer arises now the base image is pinned to a digest CI has passed on.

* Rank the PostgreSQL apt repository above the dhi.io preference

install_postgres adds the PGDG repository, but the hardened image's Package:* dhi.io preference outranks Debian's default, so apt kept dhi.io's libpq and postgresql-client-18 (which needs a newer libpq5 than dhi.io carries) failed with held broken packages. Pinning the PostgreSQL origin above the dhi.io preference lets apt take libpq from PGDG to satisfy the client.

* Use US spelling favors to pass the docs spellcheck

The docker-stack docs spellcheck rejects the British spelling favours in build.rst; use the US spelling there and in the matching install-script comments.

---------

Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
2026-10-30 13:52:53 +00:00
Jarek Potiuk 359d361de9 Base CI and PROD images on hardened Python images (#73038)
* Base CI and PROD images on hardened Python images

Squashed from 19 commits (backup branch: backup/hardened-python-base-images-pre-rebase) before rebasing onto main.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Base images on trixie and keep a legacy bookworm PROD image flavor

The 3.4.x line keeps publishing the previous images - Python compiled from sources on bookworm - as a deprecated legacy flavor, so the PROD image has to be buildable both ways while the CI image only needs the hardened trixie one. A weekly canary builds and checks the legacy images so they do not rot between releases.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Support the trixie hardened images' Debian-packaged Python and test legacy images like regular ones

The Debian 13 hardened images install Python as Debian packages under /usr instead of /opt/python, so the image has to find the base Python's prefix and the system-Python guard must not mistake those packages for Debian's own Python. The legacy canary now runs the regular PROD image tests, not just a smoke check, so the legacy images are covered as well as the default ones.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Drop CI image packages that Debian trixie no longer has

The CI image is trixie-only now: libgcc-11-dev was a bookworm-era pin (build-essential brings the matching libgcc-14-dev) and software-properties-common was removed in Debian 13; nothing uses its add-apt-repository.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Trust Microsoft's newer signing key for the trixie MSSQL repository

Microsoft signs its Debian 13 package repository with its 2025 general signing key rather than the 2015 release key, so apt on trixie rejected the repository as unsigned.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Restore the PAM configs the trixie hardened images strip

The Debian 13 hardened images ship libpam-runtime and passwd already installed but without their /etc/pam.d files, so installing them restored nothing and adding the CI image's airflow user failed with pam_start() errors.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Document the anonymous Debian package repository of the hardened images

The trixie hardened images add Docker's own package repository, whose rebuilt packages take precedence over Debian's - users extending the image should know where their packages come from, and that no credentials are needed for it.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Keep a second hardened Python out of the trixie images

On trixie, libenchant-2-dev's build dependencies need python3, which Docker's package repository satisfies with its own python-3.13 - a second interpreter in the 3.12 image that broke building C extensions. pyenchant only needs the runtime library, which it finds through the ldconfig cache, and the guard now fails early on any hardened Python other than the image's own.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Install the Python build dependencies for the legacy image flavor

Dropping the packages that only existed to compile Python left the legacy flavor compiling a Python without sqlite3, lzma or readline - CPython builds those modules only when their headers are present, and the image could not even run airflow. The build now also fails when one of them is missing.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Refresh the shared library cache after apt runs in hardened images

The hardened images ship libc-bin without its dpkg trigger, so libraries installed by apt stayed invisible to ctypes.util.find_library() - the CI image's docs build failed with "The 'enchant' C library was not found". A dpkg post-invoke hook runs ldconfig after every apt run, also in images extending ours.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Warn about apt pulling a second Python into hardened images

Extending a hardened image with apt packages that depend on python3 silently installs another Python next to the image's own, and C extensions then build against the wrong one - users need to know to check before adding such packages.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Add trixie and sigstore to the docs spelling wordlist

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Count only host keys when waiting for sshd in the CI image

OpenSSH 10 in trixie prints a '# host:port' banner comment per connection to ssh-keyscan's stdout, so the line count never matched the three host keys and every test job waited until it timed out.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Invalidate the CI image mount cache for the trixie base

The uv cache restored into CI image builds held wheels compiled against bookworm's libraries - an xmlsec wheel built against bookworm's MD5-enabled libxmlsec1 failed to import on trixie, which dropped MD5. The cache key does not encode the base OS, so it has to be bumped when that changes.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Install Java that exists on trixie in the Java examples

Debian trixie ships no Java 17. The Java SDK e2e image keeps Java 17 for Spark 3.5 by taking it from the Temurin repository, and the docs example uses trixie's Java 21.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Let old C extensions build in the trixie CI image

GCC 14 turns incompatible pointer types, integer conversions and implicit declarations into errors, and the hardened Python passes -Werror=implicit-function-declaration on to extensions - the lowest-dependency tests' old sdists (xmlsec 1.3.14, ibm-db 3.2.0) only built because those were warnings.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Document the stricter C extension builds of the hardened images

Users extending the image with old source distributions hit the same GCC 14 errors as CI, and need to know how to relax them.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Export KinD logs when the lang-SDK K8S test setup fails

A failing lang-SDK helm upgrade raised past the log export that every other step of the complete K8S test run does, so the cluster was deleted with nothing saved and the job's KinD logs artifact stayed empty.

Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x

* Keep apt from replacing the hardened images' Python

Debian 13's python3 packages conflict with the hardened images' python-3.X
packages, so installing NodeSource's nodejs (which depends on python3) into
the lang-SDK runtime image made apt remove the image's Python 3.11, and the
migration pods crash-looped. The image now holds its Python packages, so such
an install fails instead, and the runtime image copies the node binary from
the official node image rather than installing the deb.

Generated-by: Claude Opus 5

* Keep perl-base in step with the perl apt can install

The hardened apt repository publishes git, which depends on perl, but not perl
itself, while the hardened images ship their own build of perl-base. Debian's
perl depends on exactly its own perl-base version, so apt refused to install git
and the hardened CI image build failed. The image now moves perl-base to the
version of the installable perl before installing its packages; when the
hardened repository publishes a matching perl, the versions agree and nothing
changes.

Generated-by: Claude Opus 5

* Mirror the hardened base images only from main

The mirror is the default BASE_IMAGE of every Airflow build, and the job holds packages:write plus the Docker Hub credentials. Scheduled runs are always on main, but anyone with write access could dispatch the workflow from another branch and replace the mirrored images.

Generated-by: Claude Opus 5

* Check the Adoptium signing key fingerprint in the Java e2e image

The key is downloaded over HTTPS at build time; pinning its fingerprint means a replaced key on the server fails the build instead of being trusted for every package from that repository.

Generated-by: Claude Opus 5

* Allow mirroring the hardened base images from v3-*-test branches

The release branches build images from the same mirror, so maintainers need to refresh it from there too.

Generated-by: Claude Opus 5
2026-10-20 15:34:13 +02:00
0a4b5d8e49 [main] Upgrade important CI environment (#73822)
* [main] CI: Upgrade important CI environment

* Fix "prek run update-providers-build-files --all-files"

* Fix CI breakages from the dependency upgrade

- snowflake: snowflake-sqlalchemy (up to 1.11.1) cannot be imported on
  SQLAlchemy 2.1, which made ORMSelectCompileState private. Cap
  sqlalchemy<2.1 in the provider; the workspace lock resolves back to
  SQLAlchemy 2.0.54. Removing the cap is tracked in #75118.
- airflow-core: type the dag_display_name and duration hybrid expressions
  against the class (type[...] -> ColumnElement) and fix _get_dialect's
  return type, so mypy holds on SQLAlchemy 2.0 and 2.1.
- common.ai: declare the OpenAI Decisions and System One vendors added in
  pydantic-ai 2.53/2.55; reject sqlglot >= 30.22's exp.ListStage
  (Snowflake LIST/LS @stage) on the allow_writes path; adapt tests to
  pydantic-ai's streamed Anthropic requests and new request counting, and
  to sqlglot's MySQL "--+" tokenization.

Generated-by: Claude Opus 5

* Type Backfill.duration expression against the class

#75078 dropped the type: ignores in this expression, which only held
while mypy did not check the body; typing cls as type[Backfill] keeps it
clean on SQLAlchemy 2.0 and 2.1, as dag_display_name and
DagRun.duration already are. Also annotate the Anthropic stream fixture
so mypy can index its events.

Generated-by: Claude Opus 5

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Wei Lee <dev@wei-lee.me>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
2026-10-20 15:26:00 +08:00
Shahar EpsteinandJarek Potiuk 9cc55f2ad3 Update docs for the Python 3.11 baseline (#74159)
* Update docs for the Python 3.11 baseline

Contributing docs, Breeze docs, Docker stack docs and the user-facing
installation pages used Python 3.10 as the example and default
version. Examples and defaults now use 3.11; historical changelog
entries and ADRs keep their original wording.

* Re-pad the image naming table after dropping Python 3.10

* Fix stale Python 3.10 examples left in the Python 3.11 docs update

Review found commands that still pin a 3.10.x interpreter, which can no
longer install Breeze or Airflow, shell prompts and pytest transcripts
that still show 3.10, and a note that tied the supported Python range to
Airflow 3.2.0 even though the 3.2 to 3.4 release lines still accept 3.10.
Two Breeze examples used options that do not exist. The python-client
release notes presented `breeze --python 3.11` as equivalent to
`UV_PYTHON=3.11`, but the reproducibility guard checks the interpreter
Breeze itself runs on, so only the `UV_PYTHON` form works. The Docker
stack build examples now refer to the default Python version instead of
a hard-coded one, so they stop drifting with every bump.

* Fix remaining Python 3.10 references in docs and image examples

Review found more shell prompts that still show a 3.10 Breeze image.
The air-gapped image example pinned Airflow 2.5.3, which publishes no
3.11 constraints, so its download step fetched a 404 page and failed;
it also used extras that Airflow 3 no longer defines. The contributor
quick start called 3.10 end-of-life before its October 2026 end-of-life
date.

* Update Python version in database setup guide

Updated Python version in setup instructions from 3.8.10 to 3.11.16.

* Fix stale Python 3.10 constraints examples in INSTALL

Generated-by: Claude Opus 5

---------

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
2026-10-06 23:20:50 +02:00
github-actions[bot] 0e6ad21034 [main] CI: Upgrade important CI environment (#70501)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-08-13 16:27:29 +02:00
Andrew Chang 899ce98b5c Add docker stack docs example for venv scene (#69088) 2026-06-29 06:38:29 +02:00
Rahul Vats 5aa3ff7d17 Upgrade the Airflow version to 3.4.0 and Task SDK version to 1.4.0 (#69085) 2026-06-27 15:56:36 +01:00
github-actions[bot] 5f8d7400a8 [main] CI: Upgrade important CI environment (#68436)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-06-12 17:38:06 +02:00
Christos Bisias 6957d9da57 upgrade the airflow-core version to 3.3.0 and task-sdk version to 1.3.0 (#65317) 2026-04-15 19:14:25 +01:00
DaveT1991andCopilot d212f624cf Sync default image Python version references (#64994)
* Sync default image Python version references

* Fix mypy type error in _read_global_constants_assignment return type

Since ast.literal_eval() returns Any, annotating the private helper as
returning str | list[str] caused mypy errors in both callers: assigning
the result to `list[str]` and returning it as `str`. Changing the return
type to Any (which matches ast.literal_eval) resolves both without
requiring casts.

* Update scripts/ci/prek/common_prek_utils.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-04-11 11:10:00 +02:00
Jarek Potiuk 54acc0f2d7 CI: Upgrade important CI environment (#62792) 2026-03-03 16:51:33 +01:00
Jarek Potiuk eb968372fc Allow to use short SPDX licence identifier for selected files (#62073)
While we have some discussion on-going whether we should use some
shorter, machine-readable friendly versions of licence specification
in our source code headers here [1], the notion is that:

a) PMC can make judgment calls when to include different versions of
   the licence

b) This expectation only applies to the code we actually release
   in our official releases.

This change makes some judgment call on using much shorter, SPDX
driven licence headers in some specific files:

* markdown files that are intended to be consumed by agents
  (AGENTS.md, SKILLS.md, CLAUDE.md and so on)

* all the markdown .github/* files that are clearly meta-data for
  GitHub and which we exclude from released sources

We also make sure all those files are excluded from the official
source releases and distribution packages we prepare.

[1] https://lists.apache.org/thread/j1tn63r2lf13v3d1tnnqff8fkcl4nx53
2026-02-17 23:38:20 +01:00
Jens Scheffler ac9e4be2c9 Fix mpi4py building in CI (#61581) 2026-02-07 12:23:20 +01:00
Guan-Ming (Wesley) Chiu e70ac0f6bd Bump mpi4py to 4.1.0 in build-essential extend example Dockerfile (#61575)
Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>
2026-02-07 09:45:54 +02:00
Brandon Kaplan d0bb4ad34d Update Dockerfile (#59569)
grammar fix in comment
2025-12-17 16:42:59 +01:00
Jarek Potiuk f8f976dffc Remove MySQL client (#57146)
As discussed in https://lists.apache.org/thread/cwfmqbzxsm0gobtpo8kmfr99nfv29c2y
we are temporarily (or not) removing MySQL client support from Airflow
container images in order to stop our CI canary builds from failing.

If consensus or vote will be reached to remove it, we will leave it as
is, if we will find other ways how to keep mysql client support, we
will revert that change and restore MySQL client support.
2025-10-23 14:07:28 +02:00
Amogh Desai cc16b1ab71 Bump python 3.9 to 3.9.23 and 3.12 to 3.12.12 (#56505) 2025-10-09 11:43:16 -07:00
Kaxil Naik c22df6d805 Make the main branch target to Airflow 3.2 & Task SDK 1.2 (#55493)
Since 3.1 release branch is out and we are in beta phase, let's continue development in main for the next minor release.
2025-09-11 10:07:59 -06:00
Aritra BasuandJarek Potiuk 55834a7f1f Migrated prod builds to use python built from source (#53770)
Moved the prod build to also use the src built python
as done by CI. This let's us iterate faster on our
python versions without needing to wait for the community
image of python.

This involves a number of changes:

* 3.0.5 -> 3.0.6 airflow version change
* use released official packages rather than git repo to install Python
* added wget (it was added in the original image to pull python packages so
  added for compatibility) added those flags to python build (same as in the
* original build)
   * --with-ensurepip --build="$gnuArch"
   * --enable-loadable-sqlite-extension
   * --enable-option-checking=fatal
   * --enable-shared
   *  --with-lto
* added cleanup of apt after installing packages
* added removal of .pyc/.test etc. files (saves 350 MB)
* added relinking of symbolic links from /usr/python/bin to /usr/local/bin in
  the "main" image as well as in the build image.
* we do not need AIRFLOW_SETUPTOOLS_VERSION any more - this was only added to
  upgrade setuptools, because the Python official image had a very old
  setuptools version.
* checked all "customize" scripts and make them "work"
* Updated the 'version upgrade" script to upgrade AIRFLOW_PYTHON_VERSION everywhere
* Updated Changelog and documentation to update new ways of building images
* removed installation with GitHub URL (it won't work easily after splitting
  to multiple packages - not easily at least) and it's not needed
* all python, pip and similar links are created in /usr/python/bin
* /usr/python/bin is always first in the PATH - before /usr/local/bin
* added changelog entry explaining that Python's installation home has been
  moved to /usr/python/ from /usr/local
* removal of installed editable distributions in breeze happens now first and
  THEN we install when --use-airflow-version is used.
* LD_LIBRARY_PATH was not set so the shared python libraries could not be
  loaded when venv was created

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
2025-08-30 16:48:30 +02:00
Jens Scheffler ca264ba696 Clean some leftovers of Python 3.9 removal - All the rest (#52432)
* Clean some leftovers of Python 3.9 removal - All the rest

* Fix static checks
2025-06-29 17:29:46 +02:00
Jarek Potiuk a497e7c633 Remove .dev0 suffix from airflow version (#49621) 2025-04-23 22:44:32 +02:00
Kaxil Naik c14bc9b4d9 Bump Airflow to 3.1.0.dev0 (#49119)
Airflow main is now targeting 3.1 with the final rc
2025-04-22 20:32:02 +05:30
Jarek Potiuk 45cb7f1177 Remove sqlite provider from preinstalled and use aiosqlite as core dep (#48966)
Aiosqlite has never been used by the sqlite provider it is really
needed as async extension in case Airflow would be configured
with async engine. So if we are removing sqlite provider as
preinstalled, we should move aiosqlite as required depdendency of
core as otherwise creating async sqlite engine will fail.
2025-04-09 01:08:31 -04:00
Jarek Potiuk c68f31d3db Remove dev0 suffix from Airlfow version (#48934)
As we are preparing to the final release and plan to release
providers like git or fab that have Airflow 3 as minimum version
we should remove the dev0 suffix from Airflow
2025-04-08 10:15:54 -04:00
Jarek Potiuk d4473555c0 Simplify tooling by switching completely to uv (#48223)
The lazy consensus decision has been made at the devlist to switch
entirely to `uv` as development tool:

link: https://lists.apache.org/thread/6xxdon9lmjx3xh8zw09xc5k9jxb2n256

This PR implements that decision and removes a lot of baggage connected
to using `pip` additionally to uv to install and sync the environment.
It also introduces more consistency in the way how distribution
packages are used in airflow sources - basicaly switching all internal
distributions to use `pyproject.toml` approach and linking them all
together via `uv`'s workspace feature.

This enables much more streamlined development workflows, where any
part of airflow development is manageable using `uv sync` in the right
distribution - opening the way to moving more of the "sub-worfklows"
from the CI image to local virtualenv environment.

Unfortunately, such change cannot be done incrementally, really, because
any change in the project layout drags with itself a lot of changes
in the test/CI/management scripts, so we have to implement one big
PR covering the move.

This PR is "safe" in terms of the airflow and provider's code - it
does not **really** (except occasional imports and type hint changes
resulting from better isolation of packages) change Airflow code nor
it should not affect any airflow or provider code, because it does
not move any of the folder where airflow or provider's code is modified.

It does move the test code - in a number of "auxiliary" distributions
we have. It also moves the `docs` generation code to `devel-common`
and introduces separate conf.py files for every doc package.

What is still NOT done after that move and will be covered in the
follow-up changes:

* isolating docs-building to have separate configuraiton for docs
  building per distribution - allowing to run doc build locally
  with it's own conf.py file

* moving some of the tests and checks out from breeze container
  image up to the local environment (for example mypy checks) and
  likely isolating them per-provider

* Constraints are still generated using `pip freeze` and automatically
  managed by our custom scripts in `canary` builds - this will be
  replaced later by switching to `uv.lock` mechanism.

* potentially, we could merge `devel-common` and `dev` - to be
  considered as a follow-up.

* PROD image is stil build with `pip` by default when using
  `PyPI` or distribution packages  - but we do not support building
  the source image with `pip` - when building from sources, uv
  is forced internally to install packages. Currently we have
  no plans to change default PROD building to use `uv`.

This is the detailed list of changes implemented in this PR:

* uv is now mandatory to install as pre-requisite in order to
  develop airflow. We do not support installing airflow for
  development with `pip` - there will be a lot of cases where
  it will not work for development - including development
  dependencies and installing several distributions together.

* removed meta-package `hatch_build.py' and replacing it with
  pre-commit automatically modifying declarative pyproject.toml

* stripped down `hatch_build_airflow_core.py` to only cover custom
  git and asset build hooks (and renaming the file to `hatch_build.py`
  and moving all airflow dependencies to `pyproject.toml`

* converted "loose" packages in airflow repo into distributions:
  * docker-tests
  * kubernetes-tests
  * helm-tests
  * dev (here we do not have `src` subfolder - sources are directly
    in the distribution, which is for-now inconsistent with other
    distributions).

  The names of the `_tests` distribution folders have been renamed to
  the `-tests` convention to make sure the imports are always
  referring to base of each distribution and are not used from the
  content root.

* Each eof the distributions (on top of already existing airflow-core,
  task-sdk, devel-common and 90+providers has it's own set of
  dependencies, and the top-level meta-package workspace root brings
  those distributions together allowing to install them all tegether
  with a simple `uv sync --all-packages` command and come up with
  consistent set of dependencies that are good for all those
  packages (yay!). This is used to build CI image with single
  common environment to run the tests (with some quirks due to
  constraints use where we have to manually list all distributions
  until we switch to `uv.lock` mechanism)

* `doc` code is moved to `devel-common` distribution. The `doc` folder
  only keeps README informing where the other doc code is, the
  spelling_wordlist.txt and start_docs_server.sh. The documentation is
  generated in `generated/generated-docs/` folder which is entirely
  .gitignored.

* the documentation is now fully moved to:
  * `airflow-core/docs` - documentation for Airflow Core
  * `providers/**/docs` - documentation for Providers
  * `chart/docs` - documentation for Helm Chart
  * `task-sdk/docs` - documentation for Task SDK (new format not yet published)
  * `docker-stack-docs` - documentation for Docker Stack'
  * `providers-summary-docs` - documentation for provider summary page

* `versions` are not dynamically retrieved from `__init__.py` all
  of them are synchronized directly to pyproject.toml files - this
  way - except the custom build hook - we have no dynamic components
  in our `pyproject.toml` properties.

* references to extras were removed from INSTALL and other places,
  the only references to extras remains in the user documentation - we
  stop using extras for local development, we switch to using
  dependency groups.

* backtracking command was removed from breeze - we did not need it
  since we started using `uv`

* internal commands (except constraint generation) have been moved to
  `uv` from `pip`

* breeze requires `uv` to be installed and expects to be installed by
  `uv tool install -e ./dev/breeze`

* pyproject.tomls are dynamically modified when we add a version
  suffix dynamically (`--version-suffix-for-pypi`) - only for the
  time of building the versions with updated suffix

* `mypy` checks are now consistently used across all the different
  distributions and for consistency (and to fix some of the issues
  with namespace packages) rather than using "folder" approach
  when running mypy checks, even if we run mypy for whole
  distribution, we run check on individual files rather than on
  a folder. That adds consistency in execution of mypy heursistics.
  Rather than using in-container mypy script all the logic of
  selection and parameters passed to mypy are in pre-commit code.
  For now we are still using CI image to run mypy because mypy is
  very sensitive to version of dependencies installed, we should
  be able to switch to running mypy locally once we have the
  `uv.lock` mechanism incorporated in our workflows.

* lower bounds for dependencies have been set consistently across
  all the distributions. With `uv sync` and dependabot, those
  should be generally kept consistently for the future

* the `devel-common` dependencies have been groupped together in
  `devel-common` extras - including `basic`, `doc`, `doc-gen`, and
  `all` which will make it easier to install them for some OS-es
  (basic is used as default set of dependencies to cover most
  common set of development dependencies to be used for development)

* generated/provider_dependencies.json are not committed to the
  repository any longer. They are .gitignored and geberated
  on-the-flight as needed (breeze will generate them automatically
  when empty and pre-commit will always regenerate them to be
  consistent with provider's pyproject.toml files.

* `chart-utils` have been noved to `helm-tests` from `devel-common`
  as they were only used there.

* for k8s tests we are using the `uv` main `.venv` environment
  rather than creating our own `.build` environment and we use
  `uv sync` to keep it in sync

* Updated `uv` version to 0.6.10

* We are using `uv sync` to perform "upgrade to newer depencies"
  in `canary` builds and locally

* leveldb has been turned into "dependency group" and removed from
  apache-airflow and apache-airflow-core extras, it is now only
  available by google provider's leveldb optional extra to install
  with `pip`
2025-04-02 13:11:13 +02:00