* [main] CI: Upgrade important CI environment
* Ignore pydantic-ai's generic OpenAI-compatible provider in vendor drift checks
pydantic-ai-slim 2.63.0 adds a provider for any OpenAI-compatible endpoint. It needs a service name at construction and infer_model has no prefix for it, so a common.ai connection can never reach it and it is not a vendor the services page should list.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* [main] CI: Upgrade important CI environment
* Bump pypdf to 6.21.4 to fix high-severity advisories
The CI environment upgrade resolved pypdf 6.21.3, which the dependency
review gate rejects (GHSA-mfr5-77xh-9j7h, GHSA-cggf-277g-3q54,
GHSA-5849-834g-hvw5). The fixed 6.21.4 is still inside the 4-day
exclude-newer window, so add a temporary per-package cutoff for it.
Generated-by: Claude Opus 5
* Support pydantic-ai 2.60+ and opensandbox 1.2 in common.ai provider
pydantic-ai 2.60 records a durable streamed model request as the bare
response rather than a response-plus-events wrapper, so replaying or
tracking a streamed request in AirflowDurability failed to decode it.
The same release answers a tool retry with the call's own ToolReturnPart
(outcome "retried") instead of RetryPromptPart, which broke tests that
told retries from results by part type.
opensandbox 1.2 allows an egress rule with ports and no target. When the
enforced policy did not match the requested one, sorting the allowed
targets for the error message raised TypeError on that None, hiding the
real reason the sandbox was refused.
Both work with the older releases too, which the provider still supports.
Generated-by: Claude Opus 5
* Fix provider tests and type checks after the dependency upgrade
The tool-part predicates in the common.ai tests returned a plain bool, so
mypy could not narrow the message part they matched and the providers
type check failed.
On Python 3.14 the upgrade moves snowflake-connector-python to 5.1 and
snowflake-sqlalchemy to 2.1. connector 5.1's connect() no longer goes
through SnowflakeConnection, so patching that class stopped intercepting
the connection and the test tried a real login. snowflake-sqlalchemy
2.0.0 started percent-encoding "?" in the URL password, which the URI
test pinned to the older, unencoded form. Older Python versions keep the
earlier releases, so both tests have to accept either.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* Refresh the hardened base image mirror every six hours
Docker rebuilds the hardened images in place and moves the dhi.io apt
repository on at the same time. With a weekly mirror, the base images
Airflow builds on lag the repository by up to a week, and any apt
install whose dependencies were rebuilt in between fails: build-essential
currently cannot be installed because the mirrored base has an older
libbz2 than the bzip2 the repository offers
(docker-hardened-images/catalog#689). Mirroring copies manifests between
registries without pulling layers, so running it several times a day is
cheap.
Generated-by: Claude Opus 5
* Pin the hardened base images to digests the CI has passed on
Building on the mirror's moving tags made CI non-reproducible: any
refresh of the mirror changed the image every build started from,
without a change in the repository and without the full CI having run
on it. A CI run could pass or fail depending only on when it ran, and
rebuilding the same commit later gave a different image.
The images now build on a digest, and only the upgrade check moves it,
in a PR that must pass the full CI before the new base is used. The
mirror refreshes every six hours and triggers that upgrade check when it
moves a tag past its pin, so the pins follow Docker's rebuilds - and the
dhi.io apt repository moving with them - within hours rather than drift
behind it. Every pinned and mirrored digest keeps a tag of its own, so a
pinned image never becomes untagged.
The hardened images keep stock Debian's apt sources beside dhi.io, and
apt took whichever offered the higher version, mixing Debian builds into
an image whose dependencies were built by Docker. dhi.io is now
preferred whenever it publishes the package.
Generated-by: Claude Opus 5
* Let dhi.io's package versions win over the image's own
Docker renumbered its builds ("1.0.8-6+dhi2" became
"1.0.8-6+deb13u0+dhi2", which sorts lower) and its apt repository moved
to the new numbers before its images did. Every package depending on a
renumbered build - bzip2, libffi-dev, libgpg-error-dev among them - is
then uninstallable, because apt will not replace the image's
higher-numbered copy with the repository's lower-numbered one, and the
CI image build fails (docker-hardened-images/catalog#689). Every package
in the image comes from dhi.io, so letting its repository decide their
versions is safe, and it keeps the build working through such a
renumbering.
Generated-by: Claude Opus 5
* Move the base images to Docker's current Python patch releases
Docker stops rebuilding a Python patch release once it publishes the
next, so the 3.11.17, 3.12.15 and 3.13.16 images still carry the old
package numbering that the dhi.io apt repository has moved past, and
installing packages on them fails (docker-hardened-images/catalog#689).
Docker fixed it in the images of the current patch releases.
Generated with the Python part of the upgrade check, which also pins
the new images to the digests the mirror serves for them.
Generated-by: Claude Opus 5
* Compare the registry host exactly in the mirror test
Generated-by: Claude Opus 5
* Lower the dhi.io apt pin below 1000 so dependencies can resolve
At priority 1001 the Package:* dhi.io preference forces dhi.io's build even as a downgrade, so apt keeps dhi.io's libpq5 17.11 and postgresql-client-18 (which needs libpq5 >= 18.6 from PGDG) cannot be installed. Below 1000 the preference still favours dhi.io for the packages it publishes but lets apt take a newer version from another repository to satisfy a dependency. The forced "downgrade" the >1000 priority handled no longer arises now the base image is pinned to a digest CI has passed on.
* Rank the PostgreSQL apt repository above the dhi.io preference
install_postgres adds the PGDG repository, but the hardened image's Package:* dhi.io preference outranks Debian's default, so apt kept dhi.io's libpq and postgresql-client-18 (which needs a newer libpq5 than dhi.io carries) failed with held broken packages. Pinning the PostgreSQL origin above the dhi.io preference lets apt take libpq from PGDG to satisfy the client.
* Use US spelling favors to pass the docs spellcheck
The docker-stack docs spellcheck rejects the British spelling favours in build.rst; use the US spelling there and in the matching install-script comments.
---------
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
* Stop parallel prek hooks racing on __pycache__ cleanup
prek runs hooks in parallel and every breeze invocation removes all
__pycache__ directories in the checkout on start-up. Any other hook
process importing from the source tree (breeze itself, prek hook
scripts, local mypy runs) can write a fresh .pyc between rmtree emptying
a directory and removing it, which fails the breeze hook with
"OSError: [Errno 39] Directory not empty" - a flaky Static checks
failure in CI.
Redirecting bytecode of every hook into a hidden, gitignored .build/
directory keeps the source tree free of new .pyc files while hooks run,
and a new check keeps future hooks from reintroducing the race. The
cleanup additionally tolerates ENOTEMPTY for writers prek does not
control, such as an IDE.
Generated-by: Claude Opus 5
* Set PYTHONPYCACHEPREFIX on the hooks added since the branch was cut
Eight hooks that landed on main after this branch was created did not redirect their bytecode, so the new check-prek-hooks-pycache-prefix hook failed after the rebase.
Generated-by: Claude Opus 5
* Update scripts/ci/prek/check_prek_hooks_pycache_prefix.py
Co-authored-by: Wei Lee <hello@wei-lee.me>
---------
Co-authored-by: Wei Lee <hello@wei-lee.me>
* Require Python 3.11 in all distributions
Every pyproject.toml that declares requires-python now says >= 3.11 and
drops the Python 3.10 trove classifier, the tomli build requirement that
only existed for 3.10 and the per-version dependency markers for 3.10.
Lock files are regenerated.
With 3.10 gone, the tomli fallbacks in scripts, dev tooling and tests go
too, along with the tomli additional_dependencies of the prek hooks and
the blacken-docs py310 target. The tomllib imports that only sat inside
functions because of the fallback move to the top of their modules. The
four dev/registry and scripts/ci/prek modules left out of the earlier
UP017 pass also take their timezone.utc to datetime.UTC fixes here. The
minimum host Python checked by check-min-python-version, the default
Python of run_generate_constraints.py and the provider cut-off date
parsing (fromisoformat, which reads a trailing Z from 3.11) follow the
new baseline.
ruff infers py311 from the new requires-python, which enables the
PEP 646 star-index fix (UP044) applied here and makes two more rules
reportable: UP042 (str, Enum to StrEnum) alters what str() and
format() return for serialized enum values, and ASYNC109 means
dropping the public timeout parameter from async APIs. Both need
per-site judgement, so they go on the ignore list with that rationale.
The datamodel-codegen target moves to 3.11 too, with use-specialized-enum
turned off for the same reason: otherwise the generated Task SDK and
airflowctl enums would switch from (str, Enum) to StrEnum.
* Drop the Python 3.10 provider metadata shim
With provider distributions now requiring Python 3.11, breeze can derive
requires-python and classifiers from the default and allowed versions
again. The patch-exclusion test and the matching example in
MANAGING_PROVIDERS_LIFECYCLE.rst move from 3.10.0 to 3.11.0, the oldest
patch release CI still covers.
* Keep Breeze's locked dependency versions when dropping Python 3.10
Relocking Breeze for the Python 3.11 baseline ran while recently bumped versions were still inside the 4-day exclude-newer window, so it downgraded pyjwt (reverting #73969), prek, boto3/botocore and filelock. Relocking from the previous lock keeps them and only drops the packages Python 3.11 no longer needs.
Generated-by: Claude Opus 5
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* Drop Python 3.10 from CI, Breeze and the container images
Python 3.10 reaches end of life in October 2026 and more and more of
Airflow's dependencies have already dropped it, which makes the 3.10
leg the slowest and most fragile part of CI. This is the first layer
of the stack that moves the baseline to Python 3.11: Breeze stops
offering 3.10, the default Python for CI, Breeze and the images moves
to 3.11, and the PGP path for verifying the Python tarball goes away
because every remaining version ships Sigstore bundles.
Python 3.10 stays in the historical version lists that Breeze uses to
reason about older Airflow releases, and in the CI image restore
action that release branches still rely on. The distributions still
declare Python 3.10 in this layer; layer 7 bumps them once the code
no longer carries 3.10-only constructs. Until then the provider
metadata generator keeps its own 3.10-inclusive version list, and the
prek minimum-Python check keeps 3.10 because its script header still
pins it to a 3.10 interpreter until layer 2.
The shared-distribution and scripts test jobs run uv without an
interpreter pin, so uv picks the runner's system Python, which is 3.10
on the current images. Pinning them to the default CI Python keeps them
on a supported interpreter regardless of the runner image.
The image build speed check builds the CI image from the registry cache
within a fixed budget. A PR that changes install_os_dependencies.sh or
the embedded GPG keys misses the cache for the layer that compiles
Python from source. That compile took about 4.5 minutes for Python 3.10
but 11 to 18 minutes for 3.11 and newer on hosted runners, so the
budget is raised to fit a cold Python 3.11+ build.
* Explain when to drop a Python version from the CI version list
The comment above ALL_PYTHON_MAJOR_MINOR_VERSIONS described only keeping
dropped versions so cherry-picks to older branches stay tested. Python
3.10 is removed from the list because the 3.10 drop is also cherry-picked
to v3-4-test, so no release branch that still receives cherry-picks from
main supports 3.10. The comment now spells out both phases so the next
version drop follows the same rule.
The Docker image changelog gets its Python 3.10 entry under Airflow 3.4.0
here, next to the image change itself, as was done for Python 3.9.
* [main] CI: Upgrade important CI environment
* Cap azure-datalake-store below 1 in the Microsoft Azure provider
azure-datalake-store 1.x removed the lib.auth API that AzureDataLakeHook
uses, so the floor bump to 1.0.1 in the CI upgrade broke the hook and
every TestAzureDataLakeHook test. Supporting 1.x is tracked in #73795.
Generated-by: Claude Opus 5
* Regenerate provider requirement tables after the floor bumps
The floor bumps changed provider pyproject.toml files without regenerating
the README.rst and docs/index.rst requirement tables derived from them,
so update-providers-build-files and sync-provider-readme failed static checks.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This reverts commit ecfa20322b.
Main is broken, so reverting this for now.
Errors:
```
azure-ai-projects 2.7.0 has requirement openai>=3.0.0, but you have openai 2.54.0.
```
* Updates our docker commands to use breeze prefix
Updates our docker commands which are called from breeze to have a
breeze prefix, this helps by allowing breeze down to use prefix
matching to cleanup.
* Updates some of the missed container names
* Undo removed comment
* Fixes breeze cleanup ut and e2e tests
Updates the UTs to only cleanup breeze prefix,
updates e2e tests to create container with breeze
prefix in the name
* Remove merge conflict
Refactor import of tomllib for Python version compatibility.
* remove merge conflict
* fixes test_bring_all_compose_projects_down_filters_unknown_by_default
* Lint fixes
* Isolate mypy prek hooks, cover all non-provider dirs, and clean up type errors
Each non-provider mypy prek hook now builds and caches its own virtualenv
at .build/mypy-venvs/<hook>/ and its own mypy cache at
.build/mypy-caches/<hook>/. UV_PROJECT_ENVIRONMENT redirects uv away from
the project's .venv, so running the hook never mutates a contributor's
regular development environment while still matching CI's frozen
dependency set. Mypy runs with --follow-imports=silent so each hook only
reports errors for files it owns; transitive code is covered by its own
hook and different venvs no longer produce divergent results on shared
code.
Adds mypy hooks for the non-provider directories that were previously
uncovered: airflow-ctl-tests, helm-tests, airflow-e2e-tests,
task-sdk-integration-tests, docker-tests, kubernetes-tests, and shared.
The mypy-shared hook iterates every shared/<dist> workspace distribution
and builds a separate venv + cache per distribution so each shared
library is type-checked against its own dependency set.
breeze down --cleanup-mypy-cache additionally removes
.build/mypy-venvs/ and .build/mypy-caches/ so all per-hook state is
wiped alongside the existing .mypy_cache and mypy-cache-volume.
Also fixes pre-existing type errors surfaced by the newly added and
cleaned-up checks: platform-specific ignores for Linux-only
os.posix_fadvise in the shared logging helper, narrower types and
type: ignore where appropriate in shared configuration/observability/
timezones/secrets_backend/secrets_masker, Liskov override markers on
the AirflowConfigParser subclass methods, and small correctness fixes
in dev/breeze and the docker-tests / kubernetes-tests helpers so the
full non-provider mypy suite runs clean on macOS and in CI.
* Move mypy prek hooks to their respective distribution configs
The new mypy hooks for airflow-ctl-tests, helm-tests, airflow-e2e-tests,
task-sdk-integration-tests, docker-tests, and kubernetes-tests now live
in each distribution's own .pre-commit-config.yaml, matching the pattern
already used by airflow-core, task-sdk, and airflow-ctl. New .pre-commit-
config.yaml files are added to distributions that didn't have one. prek
auto-discovers nested configs, so the hooks remain part of the default
check set.
mypy-dev (covers dev + scripts), mypy-devel-common, and mypy-shared stay
at the repo root: dev/scripts/devel-common don't have their own configs,
and mypy-shared iterates every shared/<dist> distribution so has no
single home.
* Split mypy-dev and mypy-scripts, each with its own pyproject.toml config
Previously the mypy-dev prek hook ran mypy against dev/ and scripts/ in
a single invocation under the dev project's virtualenv. The two now get
independent hooks — mypy-dev in dev/.pre-commit-config.yaml and
mypy-scripts in scripts/.pre-commit-config.yaml — so each can evolve its
own dependency set and check its own folder.
Copy the full [tool.mypy] section from the root pyproject.toml into both
dev/pyproject.toml and scripts/pyproject.toml so each sub-project owns its
mypy configuration. Paths inside mypy_path are rewritten from
$MYPY_CONFIG_FILE_DIR/ to $MYPY_CONFIG_FILE_DIR/../ so they still resolve
to the repo-root siblings from the sub-project location. The decorator/
outputs plugins are scoped to dev only (scripts does not author DAG code).
mypy_local_folder.py now passes --config-file <project>/pyproject.toml
when the folder maps to one of these sub-project configs, so mypy uses
the sub-project's configuration rather than the root one.
* Teach selective-checks about the new non-provider mypy hooks
Add FileGroupForCi entries and regex patterns for helm-tests,
airflow-e2e-tests, docker-tests, kubernetes-tests, scripts, and shared
Python files, then wire them into skip_prek_hooks so the corresponding
mypy-* prek hook is only kept when its folder changed:
- mypy-scripts (split off from the old combined mypy-dev)
- mypy-airflow-ctl-tests, mypy-helm-tests, mypy-airflow-e2e-tests,
mypy-task-sdk-integration-tests, mypy-docker-tests, mypy-kubernetes-tests
- mypy-shared
Update test_selective_checks.py skip-list constants and per-case inline
skip lists to include the new hooks. Targeted test cases for files under
the new-hook directories override skip-prek-hooks to leave the matching
hook out of the skip set, confirming it will run when its folder changes.
* Trim dev/scripts pyproject mypy_path to just relevant distributions
Drop the 200+ provider path entries that were blindly copied from the root
pyproject.toml. dev and scripts only import from other non-provider
workspace members, so listing every provider src/tests directory under
mypy_path just adds noise. The remaining non-provider entries cover
everything dev or scripts plausibly import from.
* Install mypy into per-hook venvs from uv.lock via a `mypy` dep group
Each non-provider distribution with a mypy prek hook now declares a
`mypy` dependency group in its pyproject.toml resolving to
`apache-airflow-devel-common[mypy]`. mypy_local_folder.py syncs each
dedicated virtualenv with `uv sync --frozen --project <X> --group mypy`
and runs mypy with `uv run --frozen --project <X> --group mypy` — so
mypy and its type stubs come from the workspace uv.lock, not from an
ephemeral `--with` overlay whose resolution is independent of the main
lockfile. uv.lock is refreshed to include the new group.
Covers airflow-core, task-sdk, airflow-ctl, devel-common, dev, scripts,
airflow-ctl-tests, helm-tests, airflow-e2e-tests, task-sdk-integration-
tests, docker-tests, kubernetes-tests, and every shared/<dist>
workspace member.
* Drop mypy_path from dev/scripts pyprojects — venv site-packages is enough
After the switch to installing mypy (and every transitive workspace
dependency) directly into each hook's virtualenv via the `mypy` dep
group, workspace packages like airflow, airflow.sdk, airflowctl,
airflow_breeze, tests_common are all available via the venv's
site-packages. mypy resolves them without needing mypy_path entries,
so drop the copied list and leave a short comment explaining why.
* Split mypy-shared into per-distribution hooks and enforce the pattern
Each shared/<dist> workspace member now owns a mypy-shared-<dist> prek hook
backed by its own shared/<dist>/.pre-commit-config.yaml. The single
mypy-shared iterator is gone — mypy_local_folder.py accepts shared/<dist>
as a first-class folder and the per-hook virtualenv now lives at
.build/mypy-venvs/shared-<dist>/ (slash in the folder name is replaced
with a dash in the venv/cache path).
Adds a new check-shared-mypy-hooks prek hook that fails when a
shared/<dist> workspace member is missing its dedicated .pre-commit-
config.yaml, printing the exact YAML to add. Selective-checks emits one
skip entry per dist, enumerated from shared/ at run time. Contributing
docs cover the two-step process for adding a new shared library.
* Pin minimum_prek_version to 0.3.4 consistently across all configs
All .pre-commit-config.yaml files now require prek >= 0.3.4 (the version
already declared by the root config). Previously the nested configs
pinned a mix of 0.2.0, 0.3.2, and 0.3.4, so a contributor could pass the
root's version check and still trip on stale subproject pins as they
moved between directories.
* Refresh uv.lock after rebase to reflect the `mypy` dep groups
The rebase onto main resolved the uv.lock conflict by taking main's
version, so `uv sync --group mypy` would fail against uv.lock until
the groups added to the per-distribution pyprojects were re-resolved.
Regenerates the lockfile to include them.
* Add explicit selective-checks test for per-shared-dist mypy hook skipping
Verifies that when a file under shared/logging/ changes, only
mypy-shared-logging is kept among the thirteen mypy-shared-* hooks;
all other shared distributions' hooks land in the skip list. Pins the
contract that the runtime enumeration over shared/*/pyproject.toml
works as intended.
* Refresh mypy docs to match the per-hook venv + --group mypy workflow
Fills in the docs that still referenced the pre-split workflow:
- AGENTS.md: mentions `mypy-shared-<dist>` per shared workspace member and
the `uv sync --group mypy` install path for mypy itself.
- scripts/ci/prek/AGENTS.md: clarifies that non-provider mypy hooks run
locally through mypy_local_folder.py (Breeze image only needed for the
providers hook).
- dev/breeze/doc/03_developer_tasks.rst: renames stale `mypy-airflow` to
`mypy-airflow-core`, and expands the cache note to cover the per-hook
virtualenvs and caches under .build/.
- dev/breeze/doc/ci/04_selective_checks.md: expands the file-group and
skip-reason lists so every new mypy hook (scripts, task-sdk, airflow-ctl,
the six test-dir hooks, and mypy-shared-<dist> enumerated at runtime)
is documented.
* Rename mypy_local_folder.py to run_mypy_full_dist_local_venv_or_breeze_in_ci.py
Updates every .pre-commit-config.yaml entry and prose references so they
point at the new script name. Two shared configs use YAML folded-scalar
entries to stay under the 110-char yamllint limit; updates the validation
script's expected template to match.
Set `exclude-newer = "4 days"` in `[tool.uv]` to only consider package
versions uploaded at least 4 days ago. This provides supply-chain security
by giving the community time to identify and remediate compromised releases.
Also downgrade pyodps requirement for Python >= 3.13 from >=0.12.6 to
>=0.12.5.1 since 0.12.6 was published within the cooldown window.
* Update python version exclusion to 3.15
* Add 3.14 metadata version classifiers and related constants
* Regenerate Breeze command help screenshots
* Assorted workarounds to fix breeze image building
- constraints are skipped entirely
- greenlet pin updated
* Exclude cassandra
* Exclude amazon
* Exclude google
* CI: Only add pydantic extra to Airflow 2 migration tests
Before this fix there were two separate issues in the migration-test setup for Python 3.14:
1. The migration workflow always passes --airflow-extras pydantic.
2. For Python 3.14, the minimum Airflow version is resolved to 3.2.0 by get_min_airflow_version_for_python.py, and apache-airflow[pydantic]==3.2.0 is not a valid thing to install.
So when constraints installation fails, the fallback path tries to install an invalid spec.
* Disable DB migration tests for python 3.14
* Enforce werkzeug 3.x for python 3.14
* Increase K8s executor test timeout for Python 3.14
Python 3.14 changed the default multiprocessing start method from 'fork' to 'forkserver' on Linux. The forkserver start method is slower because each new process must import modules from scratch rather than copying the parent's address space. This makes `multiprocessing.Manager()` initialization take longer, causing the test to exceed its 10s timeout.
* Adapt LocalExecutor tests for Python 3.14 forkserver default
Python 3.14 changed the default multiprocessing start method from
'fork' to 'forkserver' on Linux. Like 'spawn', 'forkserver' doesn't
share the parent's address space, so mock patches applied in the test
process are invisible to worker subprocesses.
- Skip tests that mock across process boundaries on non-fork methods
- Add test_executor_lazy_worker_spawning to verify that non-fork start
methods defer worker creation and skip gc.freeze
- Make test_multiple_team_executors_isolation and
test_global_executor_without_team_name assert the correct worker
count for each start method instead of assuming pre-spawning
- Remove skip from test_clean_stop_on_signal (works on all methods)
and increase timeout from 5s to 30s for forkserver overhead
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Bump dependencies to versions supporting 3.14
* Fix PROD image build failing on Python 3.14 due to excluded providers
The PROD image build installed all provider wheels regardless of Python
version compatibility. Providers like google and amazon that exclude
Python 3.14 were still passed to pip, causing resolution failures (e.g.
ray has no cp314 wheel on PyPI).
Two fixes:
- get_distribution_specs.py now reads each wheel's Requires-Python
metadata and skips incompatible wheels instead of passing them to pip.
- The requires-python specifier generation used !=3.14 which per PEP 440
only excludes 3.14.0, not 3.14.3. Changed to !=3.14.* wildcard.
* Split core test types into 2 matrix groups to avoid OOM on Python 3.14
Non-DB core tests use xdist which runs all test types in a single pytest
process. With 2059 items across 4 workers, memory accumulates until the
OOM killer strikes at ~86% completion (exit code 137).
Split core test types into 2 groups (API/Always/CLI and
Core/Other/Serialization), similar to how provider tests already use
_split_list with NUMBER_OF_LOW_DEP_SLICES. Each group gets ~1000 items,
well under the ~1770 threshold where OOM occurs.
Update selective_checks test expectations to reflect the 2-group split.
* Gracefully handle an already removed password file in fixture
The old code had a check-then-act race (if `os.path.exists` → `os.remove`), which fails when the file doesn't exist at removal time. `contextlib.suppress(FileNotFoundError)` handles this atomically — if the file is missing (never created in this xdist worker, or removed between check and delete), it's silently ignored.
* Fix OOM and flaky tests in test_process_utils
Replace multiprocessing.Process with subprocess.Popen running minimal
inline scripts. multiprocessing.Process uses fork(), which duplicates
the entire xdist worker memory. At 95% test completion the worker has
accumulated hundreds of MBs; forking it triggers the OOM killer
(exit code 137) on Python 3.14.
subprocess.Popen starts a fresh lightweight process (~10MB) without
copying the parent's memory, avoiding the OOM entirely.
Also replace the racy ps -ax process counting in
TestKillChildProcessesByPids with psutil.pid_exists() checks on the
specific PID — the old approach was non-deterministic because unrelated
processes could start/stop between measurements.
* Add prek hook to validate python_version markers for excluded providers
When a provider declares excluded-python-versions in provider.yaml,
every dependency string referencing that provider in pyproject.toml
must carry a matching python_version marker. Missing markers cause
excluded providers to be silently installed as transitive dependencies
(e.g. aiobotocore pulling in amazon on Python 3.14).
The new check-excluded-provider-markers hook reads exclusions from
provider.yaml and validates all dependency strings in pyproject.toml
at commit time, preventing regressions like the one fixed in the
previous commit.
* Update `uv.lock`
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Skip provider tests when all test directories are excluded
When running Providers[google] or Providers[amazon] on Python 3.14,
generate_args_for_pytest removes the test folders for excluded
providers, but the skip check in _run_test only triggered when the
--ignore filter itself removed something. Since the folders were
already removed upstream, the guard condition was never met, leaving
pytest with only flags and no test directories — causing it to crash
on unrecognized custom arguments.
Remove the overly strict guard so the skip fires whenever no test
directories remain in the args.
* Fix PROD image docker tests for Python-version-excluded providers
The docker tests expected all providers from prod_image_installed_providers.txt
to be present, but providers like google and amazon declare
excluded-python-versions in their provider.yaml. On Python 3.14, these
providers are correctly excluded from the PROD image at build time, but
the tests didn't account for this.
Read provider.yaml exclusions and filter expected providers/imports based
on the Docker image's Python version.
* Skip Python-incompatible provider wheels during PROD image build
get_distribution_specs.py now reads Requires-Python metadata from each
wheel and skips wheels that are incompatible with the running
interpreter. This prevents excluded providers (e.g. amazon on 3.14)
from being passed to pip/uv and installed despite their exclusion.
Also fix the requires-python specifier generation in packages.py:
!=3.14 per PEP 440 only excludes 3.14.0, not 3.14.2. Use !=3.14.*
wildcard to exclude the entire minor version.
IntelliJ IDEA .iml files could end up in sdist/wheel packages when
building distributions that lack their own .gitignore. Adding *.iml
to every distribution's .gitignore ensures these IDE metadata files
are excluded from built packages.
* Fix all build-system/requires including transitive dependencies
Add a feature to fix all build-system/requires to make build
reproducibility works. The build-system requires are managed
automatically by the `upgrade-important-versions` prek hook.
This should not only provide reproducibility, but also it should
prevent cases where transient dependency upgrade might break
building sdist (for example as it happened with virtualenv 21
breaking hatchling with https://github.com/pypa/hatch/issues/2193
* Fix mypy 'Cannot infer type of lambda' error in upgrade script
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Since UV and other installers essentially ignore < limit in the
python-requires, a good practice is to simply exclude current Python
versions that we **know** we are not compatible with rather than
using <.
Python 3.14 is a thing now and people will get it by default as
latest versions, but we do not run our tests with Python 3.14
yet (not until March/April 2026). Providers have all
upper-bind exclusion except those that do not support Python 3.13,
but that's ok because they are actually limited by Airflow they
can be installed on. But for distributions like airflow-ctl,
we should explicitly exclude 3.14 until we support and test it.
This PR synchronizes all python-requires in non-provider
distribution to consistently exclude Python 3.14 with !=.
* Implement initial integration test for airflowctl with 3.1
* password can be passed without interaction, update integration tests
* Add AIRFLOW_CLI_DEBUG_MODE for enhanced CLI debugging and update integration tests to skip keyring
* Warn user while running each command if debug mode enabled and explicitly state it shouldn't be used unless debugging or integration tests
* Move python-on-whales to devel-common, use shared docker-composer file, update documentation mistakes
* remove shared python-on-whales from airflow-ctl-tests/
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* Decouple docker compose logic from test method to pytest_sessionstart in conftest
* Move python_on_whale import to file level
* Reorder dependencies in pyproject.toml for consistency
* Add workspace to main pyproject.toml, remove unused variable, move console to singleton __init__.py
* Add workspace to main pyproject.toml, remove unused variable, move console to singleton __init__.py
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
The pre-commit is a fantastic tool, and we heavily used it for years,
but generally the tool stagnated and is not showing a sign of adapting
to our needs. For years we tried to convince pre-commit maintainers that
things like autocomplete are necessary - but it met with pretty much
resistance (if not hostility) from the maintainer. Also there was no
chance for them to accept expectations of bigger projects like ours,
where we have a huge monorepo and not only multiple needs but also
different parts of the repo needing different language support (golang,
typescript soon) - and apparenty the maintainer of pre-commit does not
think monorepo is a good thing at all. Similarly - they did not recognize
the raise of `uv` and the only way to use `uv` with pre-commit is to patch
it by installing `pre-comit-uv` that essentialy patches pre-commit with
uv support. This is not really sustainable and the tool lags behind many
of our needs.
Luckily - we have new project in town - prek - which rewrites pre-commit
that is 100% compatible (now), 10x faster (because rust), uses `uv`
natively, supports auto-complete already and they have very friendly
maintainer who is not only supporting us but also very happily works
on improving `prek` to close all the gaps, and plans to implement (with
our support of course and cooperation) monorepo support - that will
allow us to modularise our pre-commits.
This PR switches our pre-commit support to use prek exclusively:
* breeze static checks command is completely removed
* custom auto-complete code in breeze as well
* instructions are updated to setup prek instead of precommit
* CI is updated to run prek instead of pre-commmit
* documentation for static checks is reviewed and new features that
prek enables are added
* Breeze changes for Python 3.13
* Pyproject.toml provider changes for Python 3.13
* README.rst changes for providers for Python 3.13
* Add Python 3.13 support for Airflow.
Added Python 3.13 support across the codebase, including:
* CI/CD workflow updates to handle Python 3.13.
* Dockerfile and environment variable adjustments for compatibility.
* Documentation updates to list Python 3.13 as supported.
* Dependency and code changes for compatibility with new/changed
libraries (e.g., numpy 2.x, greenlet, pendulum).
* Test and plugin manager adjustments for Python 3.13-specific behaviors.
* Removal or skipping of tests and features not compatible with Python 3.13.
* Improved error handling and logging for unsupported plugins/providers
on Python 3.13.
* Updated all provider README.rst files to:
* List Python 3.13 as a supported version.
* Adjust dependency requirements for Python 3.13 (e.g., pandas,
pyarrow, ray, etc.).
* Add or update notes about excluded or conditionally supported
dependencies for Python 3.13.
* Document any known incompatibilities or workarounds for specific
providers.
* Updated all provider pyproject.toml files and automation to:
* Add Python 3.13 to the supported classifiers.
* Adjust requires-python and dependency constraints for Python 3.13.
* Exclude Python 3.13 for providers that are not yet compatible (e.g.,
those depending on Flask AppBuilder).
* Add conditional dependencies for new versions of libraries required
by Python 3.13.
* Updated Breeze and related scripts to:
* Build and test with Python 3.13.
* Update documentation and release scripts to include Python 3.13.
* Ensure all dev tooling and test environments work with Python 3.13.
* Removed remaining FAB related code from core and removed tests for
older FAB provider versions (<1.3.0)
* some of the tests in core still depend on FAB - but we make them
optional - both for parsing (we need to skip some imports) and
execution - because they test a legacy behaviour of embedding
Flask plugins that is still supported in core.
* k8s tests now run also with SimpleAuthManager and SimpleAuthManager
is used automatically when runnin tests in Python 3.13
* migration tests had to be excluded fo Python 3.13 because we cannot
migrate down on Python 3.13 without FAB tables created. The tests
will be brought back when FAB is supported or when we release 3.1
and start working on 3.2, we should be able to migrate down to 3.1
* Weaviate tests have been updated to handle case where weaviate-client
is > 2.10.0 - because it started to use httpx instead of requests,
and Python 3.13 actually unblocks using older version of weaviate
(it was previously blocked indirectly via Flask dependencies and
grpcio).
* Added doc change in our rules on what is the "default" Python version
for airflow images - handling the case where we cannot use "newest"
Python version as default where not all "regular" image providers are
supported in the latest version
* Connection is now flushed in `configure_git_connection_for_dag_bundle`
just before `clear_db_connections` - in order to handle Python 3.13
case where FAB app is not initialized by other fixtures - performing
implicit flush along the way
* Cleaning db connections was added to test_dag_run and test_dags test
in fast_api as the test implicitly rely on having connections
cleared (no test git connections added) - this will make the tests
side-effect free.
* Conditionally skipping some serialization tests involving yandex
and ray on Python 3.13 until they support Python 3.13
Follow up after #52967 -> from later discussions it turned out that
it's not really the ~= that is wrong and ambiguous, but that just
upper-binding of Python version is generally considered as a bad
idea - and it's not Astral's view but it's general consensus that
upper-binding of "python-requires" is bad. Since ~= implies
upper-binding, simply replacing it with >= is likely the best option
we can choose.
github-actions[bot]andJarek Potiuk
Shahar EpsteinandJarek Potiuk
Dev-iL
Ash Berlin-Taylor
Jens Scheffler
Aritra Basu
사재혁
Jason(Zhe-You) Liu
Wei Lee
Anusha KoviandKalyan R
Bugra OzturkandJarek Potiuk
Kaxil Naik
GPK
Tzu-ping Chung