* [main] CI: Upgrade important CI environment
* Ignore pydantic-ai's generic OpenAI-compatible provider in vendor drift checks
pydantic-ai-slim 2.63.0 adds a provider for any OpenAI-compatible endpoint. It needs a service name at construction and infer_model has no prefix for it, so a common.ai connection can never reach it and it is not a vendor the services page should list.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* [main] CI: Upgrade important CI environment
* Bump pypdf to 6.21.4 to fix high-severity advisories
The CI environment upgrade resolved pypdf 6.21.3, which the dependency
review gate rejects (GHSA-mfr5-77xh-9j7h, GHSA-cggf-277g-3q54,
GHSA-5849-834g-hvw5). The fixed 6.21.4 is still inside the 4-day
exclude-newer window, so add a temporary per-package cutoff for it.
Generated-by: Claude Opus 5
* Support pydantic-ai 2.60+ and opensandbox 1.2 in common.ai provider
pydantic-ai 2.60 records a durable streamed model request as the bare
response rather than a response-plus-events wrapper, so replaying or
tracking a streamed request in AirflowDurability failed to decode it.
The same release answers a tool retry with the call's own ToolReturnPart
(outcome "retried") instead of RetryPromptPart, which broke tests that
told retries from results by part type.
opensandbox 1.2 allows an egress rule with ports and no target. When the
enforced policy did not match the requested one, sorting the allowed
targets for the error message raised TypeError on that None, hiding the
real reason the sandbox was refused.
Both work with the older releases too, which the provider still supports.
Generated-by: Claude Opus 5
* Fix provider tests and type checks after the dependency upgrade
The tool-part predicates in the common.ai tests returned a plain bool, so
mypy could not narrow the message part they matched and the providers
type check failed.
On Python 3.14 the upgrade moves snowflake-connector-python to 5.1 and
snowflake-sqlalchemy to 2.1. connector 5.1's connect() no longer goes
through SnowflakeConnection, so patching that class stopped intercepting
the connection and the test tried a real login. snowflake-sqlalchemy
2.0.0 started percent-encoding "?" in the URL password, which the URI
test pinned to the older, unencoded form. Older Python versions keep the
earlier releases, so both tests have to accept either.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* Refresh the hardened base image mirror every six hours
Docker rebuilds the hardened images in place and moves the dhi.io apt
repository on at the same time. With a weekly mirror, the base images
Airflow builds on lag the repository by up to a week, and any apt
install whose dependencies were rebuilt in between fails: build-essential
currently cannot be installed because the mirrored base has an older
libbz2 than the bzip2 the repository offers
(docker-hardened-images/catalog#689). Mirroring copies manifests between
registries without pulling layers, so running it several times a day is
cheap.
Generated-by: Claude Opus 5
* Pin the hardened base images to digests the CI has passed on
Building on the mirror's moving tags made CI non-reproducible: any
refresh of the mirror changed the image every build started from,
without a change in the repository and without the full CI having run
on it. A CI run could pass or fail depending only on when it ran, and
rebuilding the same commit later gave a different image.
The images now build on a digest, and only the upgrade check moves it,
in a PR that must pass the full CI before the new base is used. The
mirror refreshes every six hours and triggers that upgrade check when it
moves a tag past its pin, so the pins follow Docker's rebuilds - and the
dhi.io apt repository moving with them - within hours rather than drift
behind it. Every pinned and mirrored digest keeps a tag of its own, so a
pinned image never becomes untagged.
The hardened images keep stock Debian's apt sources beside dhi.io, and
apt took whichever offered the higher version, mixing Debian builds into
an image whose dependencies were built by Docker. dhi.io is now
preferred whenever it publishes the package.
Generated-by: Claude Opus 5
* Let dhi.io's package versions win over the image's own
Docker renumbered its builds ("1.0.8-6+dhi2" became
"1.0.8-6+deb13u0+dhi2", which sorts lower) and its apt repository moved
to the new numbers before its images did. Every package depending on a
renumbered build - bzip2, libffi-dev, libgpg-error-dev among them - is
then uninstallable, because apt will not replace the image's
higher-numbered copy with the repository's lower-numbered one, and the
CI image build fails (docker-hardened-images/catalog#689). Every package
in the image comes from dhi.io, so letting its repository decide their
versions is safe, and it keeps the build working through such a
renumbering.
Generated-by: Claude Opus 5
* Move the base images to Docker's current Python patch releases
Docker stops rebuilding a Python patch release once it publishes the
next, so the 3.11.17, 3.12.15 and 3.13.16 images still carry the old
package numbering that the dhi.io apt repository has moved past, and
installing packages on them fails (docker-hardened-images/catalog#689).
Docker fixed it in the images of the current patch releases.
Generated with the Python part of the upgrade check, which also pins
the new images to the digests the mirror serves for them.
Generated-by: Claude Opus 5
* Compare the registry host exactly in the mirror test
Generated-by: Claude Opus 5
* Lower the dhi.io apt pin below 1000 so dependencies can resolve
At priority 1001 the Package:* dhi.io preference forces dhi.io's build even as a downgrade, so apt keeps dhi.io's libpq5 17.11 and postgresql-client-18 (which needs libpq5 >= 18.6 from PGDG) cannot be installed. Below 1000 the preference still favours dhi.io for the packages it publishes but lets apt take a newer version from another repository to satisfy a dependency. The forced "downgrade" the >1000 priority handled no longer arises now the base image is pinned to a digest CI has passed on.
* Rank the PostgreSQL apt repository above the dhi.io preference
install_postgres adds the PGDG repository, but the hardened image's Package:* dhi.io preference outranks Debian's default, so apt kept dhi.io's libpq and postgresql-client-18 (which needs a newer libpq5 than dhi.io carries) failed with held broken packages. Pinning the PostgreSQL origin above the dhi.io preference lets apt take libpq from PGDG to satisfy the client.
* Use US spelling favors to pass the docs spellcheck
The docker-stack docs spellcheck rejects the British spelling favours in build.rst; use the US spelling there and in the matching install-script comments.
---------
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
* Base CI and PROD images on hardened Python images
Squashed from 19 commits (backup branch: backup/hardened-python-base-images-pre-rebase) before rebasing onto main.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Base images on trixie and keep a legacy bookworm PROD image flavor
The 3.4.x line keeps publishing the previous images - Python compiled from sources on bookworm - as a deprecated legacy flavor, so the PROD image has to be buildable both ways while the CI image only needs the hardened trixie one. A weekly canary builds and checks the legacy images so they do not rot between releases.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Support the trixie hardened images' Debian-packaged Python and test legacy images like regular ones
The Debian 13 hardened images install Python as Debian packages under /usr instead of /opt/python, so the image has to find the base Python's prefix and the system-Python guard must not mistake those packages for Debian's own Python. The legacy canary now runs the regular PROD image tests, not just a smoke check, so the legacy images are covered as well as the default ones.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Drop CI image packages that Debian trixie no longer has
The CI image is trixie-only now: libgcc-11-dev was a bookworm-era pin (build-essential brings the matching libgcc-14-dev) and software-properties-common was removed in Debian 13; nothing uses its add-apt-repository.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Trust Microsoft's newer signing key for the trixie MSSQL repository
Microsoft signs its Debian 13 package repository with its 2025 general signing key rather than the 2015 release key, so apt on trixie rejected the repository as unsigned.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Restore the PAM configs the trixie hardened images strip
The Debian 13 hardened images ship libpam-runtime and passwd already installed but without their /etc/pam.d files, so installing them restored nothing and adding the CI image's airflow user failed with pam_start() errors.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Document the anonymous Debian package repository of the hardened images
The trixie hardened images add Docker's own package repository, whose rebuilt packages take precedence over Debian's - users extending the image should know where their packages come from, and that no credentials are needed for it.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Keep a second hardened Python out of the trixie images
On trixie, libenchant-2-dev's build dependencies need python3, which Docker's package repository satisfies with its own python-3.13 - a second interpreter in the 3.12 image that broke building C extensions. pyenchant only needs the runtime library, which it finds through the ldconfig cache, and the guard now fails early on any hardened Python other than the image's own.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Install the Python build dependencies for the legacy image flavor
Dropping the packages that only existed to compile Python left the legacy flavor compiling a Python without sqlite3, lzma or readline - CPython builds those modules only when their headers are present, and the image could not even run airflow. The build now also fails when one of them is missing.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Refresh the shared library cache after apt runs in hardened images
The hardened images ship libc-bin without its dpkg trigger, so libraries installed by apt stayed invisible to ctypes.util.find_library() - the CI image's docs build failed with "The 'enchant' C library was not found". A dpkg post-invoke hook runs ldconfig after every apt run, also in images extending ours.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Warn about apt pulling a second Python into hardened images
Extending a hardened image with apt packages that depend on python3 silently installs another Python next to the image's own, and C extensions then build against the wrong one - users need to know to check before adding such packages.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Add trixie and sigstore to the docs spelling wordlist
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Count only host keys when waiting for sshd in the CI image
OpenSSH 10 in trixie prints a '# host:port' banner comment per connection to ssh-keyscan's stdout, so the line count never matched the three host keys and every test job waited until it timed out.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Invalidate the CI image mount cache for the trixie base
The uv cache restored into CI image builds held wheels compiled against bookworm's libraries - an xmlsec wheel built against bookworm's MD5-enabled libxmlsec1 failed to import on trixie, which dropped MD5. The cache key does not encode the base OS, so it has to be bumped when that changes.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Install Java that exists on trixie in the Java examples
Debian trixie ships no Java 17. The Java SDK e2e image keeps Java 17 for Spark 3.5 by taking it from the Temurin repository, and the docs example uses trixie's Java 21.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Let old C extensions build in the trixie CI image
GCC 14 turns incompatible pointer types, integer conversions and implicit declarations into errors, and the hardened Python passes -Werror=implicit-function-declaration on to extensions - the lowest-dependency tests' old sdists (xmlsec 1.3.14, ibm-db 3.2.0) only built because those were warnings.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Document the stricter C extension builds of the hardened images
Users extending the image with old source distributions hit the same GCC 14 errors as CI, and need to know how to relax them.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Export KinD logs when the lang-SDK K8S test setup fails
A failing lang-SDK helm upgrade raised past the log export that every other step of the complete K8S test run does, so the cluster was deleted with nothing saved and the job's KinD logs artifact stayed empty.
Generated-by: Claude Opus 5
Claude-Session: https://claude.ai/code/session_014np1fVAxHmGaBFt3qebJ8x
* Keep apt from replacing the hardened images' Python
Debian 13's python3 packages conflict with the hardened images' python-3.X
packages, so installing NodeSource's nodejs (which depends on python3) into
the lang-SDK runtime image made apt remove the image's Python 3.11, and the
migration pods crash-looped. The image now holds its Python packages, so such
an install fails instead, and the runtime image copies the node binary from
the official node image rather than installing the deb.
Generated-by: Claude Opus 5
* Keep perl-base in step with the perl apt can install
The hardened apt repository publishes git, which depends on perl, but not perl
itself, while the hardened images ship their own build of perl-base. Debian's
perl depends on exactly its own perl-base version, so apt refused to install git
and the hardened CI image build failed. The image now moves perl-base to the
version of the installable perl before installing its packages; when the
hardened repository publishes a matching perl, the versions agree and nothing
changes.
Generated-by: Claude Opus 5
* Mirror the hardened base images only from main
The mirror is the default BASE_IMAGE of every Airflow build, and the job holds packages:write plus the Docker Hub credentials. Scheduled runs are always on main, but anyone with write access could dispatch the workflow from another branch and replace the mirrored images.
Generated-by: Claude Opus 5
* Check the Adoptium signing key fingerprint in the Java e2e image
The key is downloaded over HTTPS at build time; pinning its fingerprint means a replaced key on the server fails the build instead of being trusted for every package from that repository.
Generated-by: Claude Opus 5
* Allow mirroring the hardened base images from v3-*-test branches
The release branches build images from the same mirror, so maintainers need to refresh it from there too.
Generated-by: Claude Opus 5
* [main] CI: Upgrade important CI environment
* Fix "prek run update-providers-build-files --all-files"
* Fix CI breakages from the dependency upgrade
- snowflake: snowflake-sqlalchemy (up to 1.11.1) cannot be imported on
SQLAlchemy 2.1, which made ORMSelectCompileState private. Cap
sqlalchemy<2.1 in the provider; the workspace lock resolves back to
SQLAlchemy 2.0.54. Removing the cap is tracked in #75118.
- airflow-core: type the dag_display_name and duration hybrid expressions
against the class (type[...] -> ColumnElement) and fix _get_dialect's
return type, so mypy holds on SQLAlchemy 2.0 and 2.1.
- common.ai: declare the OpenAI Decisions and System One vendors added in
pydantic-ai 2.53/2.55; reject sqlglot >= 30.22's exp.ListStage
(Snowflake LIST/LS @stage) on the allow_writes path; adapt tests to
pydantic-ai's streamed Anthropic requests and new request counting, and
to sqlglot's MySQL "--+" tokenization.
Generated-by: Claude Opus 5
* Type Backfill.duration expression against the class
#75078 dropped the type: ignores in this expression, which only held
while mypy did not check the body; typing cls as type[Backfill] keeps it
clean on SQLAlchemy 2.0 and 2.1, as dag_display_name and
DagRun.duration already are. Also annotate the Anthropic stream fixture
so mypy can index its events.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Wei Lee <dev@wei-lee.me>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Previously to see the same error in docs build as you would get in CI,
you would have to run both the docs build and the spellchecking build.
The issue was that the spellcheck (using sphinx-contrib's spelling) was
defined as a new builder, meaning you could either build HTML, or you
can check spelling. On CI, this didn't result in any extra wall-clock
time as the jobs were parallelized (but this change does remove a job
from the overall matrix), but locally that was almost always not the
case, so this would mean full build to view the HTML, and spell checking
build to look for errors, resulting in it taking twice as long.
The upstream spellchecker was also slower than it should be mainly
because its importable-module filter calls find_spec on every distinct
identifier-like word before consulting the dictionary. Here it runs only
for words that are still misspelled. Word lists, filters and
`:spelling:ignore:` behave as before.
This also handles errors a lot more nicely (hence all the code deletions):
- They are written natively to a JSON file, not scraped from a stream
- They also show up as GH annotations (Where we can attribute it to a
real file) to make fixing them easier on PRs.
Please note: this is lightly breaking for Airflow contributors as
`breeze build-docs --docs-only` and `--spellcheck-only` are gone --
every build checks spelling now as it is essentially free (on the order
of milliseconds per module).
* Update docs for the Python 3.11 baseline
Contributing docs, Breeze docs, Docker stack docs and the user-facing
installation pages used Python 3.10 as the example and default
version. Examples and defaults now use 3.11; historical changelog
entries and ADRs keep their original wording.
* Re-pad the image naming table after dropping Python 3.10
* Fix stale Python 3.10 examples left in the Python 3.11 docs update
Review found commands that still pin a 3.10.x interpreter, which can no
longer install Breeze or Airflow, shell prompts and pytest transcripts
that still show 3.10, and a note that tied the supported Python range to
Airflow 3.2.0 even though the 3.2 to 3.4 release lines still accept 3.10.
Two Breeze examples used options that do not exist. The python-client
release notes presented `breeze --python 3.11` as equivalent to
`UV_PYTHON=3.11`, but the reproducibility guard checks the interpreter
Breeze itself runs on, so only the `UV_PYTHON` form works. The Docker
stack build examples now refer to the default Python version instead of
a hard-coded one, so they stop drifting with every bump.
* Fix remaining Python 3.10 references in docs and image examples
Review found more shell prompts that still show a 3.10 Breeze image.
The air-gapped image example pinned Airflow 2.5.3, which publishes no
3.11 constraints, so its download step fetched a 404 page and failed;
it also used extras that Airflow 3 no longer defines. The contributor
quick start called 3.10 end-of-life before its October 2026 end-of-life
date.
* Update Python version in database setup guide
Updated Python version in setup instructions from 3.8.10 to 3.11.16.
* Fix stale Python 3.10 constraints examples in INSTALL
Generated-by: Claude Opus 5
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* Require Python 3.11 in all distributions
Every pyproject.toml that declares requires-python now says >= 3.11 and
drops the Python 3.10 trove classifier, the tomli build requirement that
only existed for 3.10 and the per-version dependency markers for 3.10.
Lock files are regenerated.
With 3.10 gone, the tomli fallbacks in scripts, dev tooling and tests go
too, along with the tomli additional_dependencies of the prek hooks and
the blacken-docs py310 target. The tomllib imports that only sat inside
functions because of the fallback move to the top of their modules. The
four dev/registry and scripts/ci/prek modules left out of the earlier
UP017 pass also take their timezone.utc to datetime.UTC fixes here. The
minimum host Python checked by check-min-python-version, the default
Python of run_generate_constraints.py and the provider cut-off date
parsing (fromisoformat, which reads a trailing Z from 3.11) follow the
new baseline.
ruff infers py311 from the new requires-python, which enables the
PEP 646 star-index fix (UP044) applied here and makes two more rules
reportable: UP042 (str, Enum to StrEnum) alters what str() and
format() return for serialized enum values, and ASYNC109 means
dropping the public timeout parameter from async APIs. Both need
per-site judgement, so they go on the ignore list with that rationale.
The datamodel-codegen target moves to 3.11 too, with use-specialized-enum
turned off for the same reason: otherwise the generated Task SDK and
airflowctl enums would switch from (str, Enum) to StrEnum.
* Drop the Python 3.10 provider metadata shim
With provider distributions now requiring Python 3.11, breeze can derive
requires-python and classifiers from the default and allowed versions
again. The patch-exclusion test and the matching example in
MANAGING_PROVIDERS_LIFECYCLE.rst move from 3.10.0 to 3.11.0, the oldest
patch release CI still covers.
* Keep Breeze's locked dependency versions when dropping Python 3.10
Relocking Breeze for the Python 3.11 baseline ran while recently bumped versions were still inside the 4-day exclude-newer window, so it downgraded pyjwt (reverting #73969), prek, boto3/botocore and filelock. Relocking from the previous lock keeps them and only drops the packages Python 3.11 no longer needs.
Generated-by: Claude Opus 5
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* Drop Python 3.10 from CI, Breeze and the container images
Python 3.10 reaches end of life in October 2026 and more and more of
Airflow's dependencies have already dropped it, which makes the 3.10
leg the slowest and most fragile part of CI. This is the first layer
of the stack that moves the baseline to Python 3.11: Breeze stops
offering 3.10, the default Python for CI, Breeze and the images moves
to 3.11, and the PGP path for verifying the Python tarball goes away
because every remaining version ships Sigstore bundles.
Python 3.10 stays in the historical version lists that Breeze uses to
reason about older Airflow releases, and in the CI image restore
action that release branches still rely on. The distributions still
declare Python 3.10 in this layer; layer 7 bumps them once the code
no longer carries 3.10-only constructs. Until then the provider
metadata generator keeps its own 3.10-inclusive version list, and the
prek minimum-Python check keeps 3.10 because its script header still
pins it to a 3.10 interpreter until layer 2.
The shared-distribution and scripts test jobs run uv without an
interpreter pin, so uv picks the runner's system Python, which is 3.10
on the current images. Pinning them to the default CI Python keeps them
on a supported interpreter regardless of the runner image.
The image build speed check builds the CI image from the registry cache
within a fixed budget. A PR that changes install_os_dependencies.sh or
the embedded GPG keys misses the cache for the layer that compiles
Python from source. That compile took about 4.5 minutes for Python 3.10
but 11 to 18 minutes for 3.11 and newer on hosted runners, so the
budget is raised to fit a cold Python 3.11+ build.
* Explain when to drop a Python version from the CI version list
The comment above ALL_PYTHON_MAJOR_MINOR_VERSIONS described only keeping
dropped versions so cherry-picks to older branches stay tested. Python
3.10 is removed from the list because the 3.10 drop is also cherry-picked
to v3-4-test, so no release branch that still receives cherry-picks from
main supports 3.10. The comment now spells out both phases so the next
version drop follows the same rule.
The Docker image changelog gets its Python 3.10 entry under Airflow 3.4.0
here, next to the image change itself, as was done for Python 3.9.
* [main] CI: Upgrade important CI environment
* Cap azure-datalake-store below 1 in the Microsoft Azure provider
azure-datalake-store 1.x removed the lib.auth API that AzureDataLakeHook
uses, so the floor bump to 1.0.1 in the CI upgrade broke the hook and
every TestAzureDataLakeHook test. Supporting 1.x is tracked in #73795.
Generated-by: Claude Opus 5
* Regenerate provider requirement tables after the floor bumps
The floor bumps changed provider pyproject.toml files without regenerating
the README.rst and docs/index.rst requirement tables derived from them,
so update-providers-build-files and sync-provider-readme failed static checks.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This reverts commit ecfa20322b.
Main is broken, so reverting this for now.
Errors:
```
azure-ai-projects 2.7.0 has requirement openai>=3.0.0, but you have openai 2.54.0.
```
Record in the 'Changes after publishing the images' table that the 3.3.0 Python 3.11/3.12 images had their downgraded databricks stack corrected (provider 6.13.0 -> 7.16.1, connector 2.0.2 -> 4.2.5, thrift 0.23.0 -> 0.16.0).
related: #69603
#65374 added opensearch to prod_image_installed_providers.txt (so the PROD
image verify asserts it is installed) but did not add it to the default
image extras. opensearch is therefore the only required provider that is
neither a default extra nor a bundled core dependency, so its only path
into the image was being built from source.
On a release branch the source build skips already-released providers
(the final tag providers-opensearch/<v> exists), so opensearch is the
single required provider with no fallback and the PROD image verify
(test_required_providers_are_installed) fails. main passes only because
its dev-mode build compiles every provider from source in-place.
Add opensearch to the default extras so it is backfilled from PyPI the
same way every other required provider is, making the PROD image
consistent across dev and release builds.
* Sync default image Python version references
* Fix mypy type error in _read_global_constants_assignment return type
Since ast.literal_eval() returns Any, annotating the private helper as
returning str | list[str] caused mypy errors in both callers: assigning
the result to `list[str]` and returning it as `str`. Changing the return
type to Any (which matches ast.literal_eval) resolves both without
requiring casts.
* Update scripts/ci/prek/common_prek_utils.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Set `exclude-newer = "4 days"` in `[tool.uv]` to only consider package
versions uploaded at least 4 days ago. This provides supply-chain security
by giving the community time to identify and remediate compromised releases.
Also downgrade pyodps requirement for Python >= 3.13 from >=0.12.6 to
>=0.12.5.1 since 0.12.6 was published within the cooldown window.
* Update python version exclusion to 3.15
* Add 3.14 metadata version classifiers and related constants
* Regenerate Breeze command help screenshots
* Assorted workarounds to fix breeze image building
- constraints are skipped entirely
- greenlet pin updated
* Exclude cassandra
* Exclude amazon
* Exclude google
* CI: Only add pydantic extra to Airflow 2 migration tests
Before this fix there were two separate issues in the migration-test setup for Python 3.14:
1. The migration workflow always passes --airflow-extras pydantic.
2. For Python 3.14, the minimum Airflow version is resolved to 3.2.0 by get_min_airflow_version_for_python.py, and apache-airflow[pydantic]==3.2.0 is not a valid thing to install.
So when constraints installation fails, the fallback path tries to install an invalid spec.
* Disable DB migration tests for python 3.14
* Enforce werkzeug 3.x for python 3.14
* Increase K8s executor test timeout for Python 3.14
Python 3.14 changed the default multiprocessing start method from 'fork' to 'forkserver' on Linux. The forkserver start method is slower because each new process must import modules from scratch rather than copying the parent's address space. This makes `multiprocessing.Manager()` initialization take longer, causing the test to exceed its 10s timeout.
* Adapt LocalExecutor tests for Python 3.14 forkserver default
Python 3.14 changed the default multiprocessing start method from
'fork' to 'forkserver' on Linux. Like 'spawn', 'forkserver' doesn't
share the parent's address space, so mock patches applied in the test
process are invisible to worker subprocesses.
- Skip tests that mock across process boundaries on non-fork methods
- Add test_executor_lazy_worker_spawning to verify that non-fork start
methods defer worker creation and skip gc.freeze
- Make test_multiple_team_executors_isolation and
test_global_executor_without_team_name assert the correct worker
count for each start method instead of assuming pre-spawning
- Remove skip from test_clean_stop_on_signal (works on all methods)
and increase timeout from 5s to 30s for forkserver overhead
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Bump dependencies to versions supporting 3.14
* Fix PROD image build failing on Python 3.14 due to excluded providers
The PROD image build installed all provider wheels regardless of Python
version compatibility. Providers like google and amazon that exclude
Python 3.14 were still passed to pip, causing resolution failures (e.g.
ray has no cp314 wheel on PyPI).
Two fixes:
- get_distribution_specs.py now reads each wheel's Requires-Python
metadata and skips incompatible wheels instead of passing them to pip.
- The requires-python specifier generation used !=3.14 which per PEP 440
only excludes 3.14.0, not 3.14.3. Changed to !=3.14.* wildcard.
* Split core test types into 2 matrix groups to avoid OOM on Python 3.14
Non-DB core tests use xdist which runs all test types in a single pytest
process. With 2059 items across 4 workers, memory accumulates until the
OOM killer strikes at ~86% completion (exit code 137).
Split core test types into 2 groups (API/Always/CLI and
Core/Other/Serialization), similar to how provider tests already use
_split_list with NUMBER_OF_LOW_DEP_SLICES. Each group gets ~1000 items,
well under the ~1770 threshold where OOM occurs.
Update selective_checks test expectations to reflect the 2-group split.
* Gracefully handle an already removed password file in fixture
The old code had a check-then-act race (if `os.path.exists` → `os.remove`), which fails when the file doesn't exist at removal time. `contextlib.suppress(FileNotFoundError)` handles this atomically — if the file is missing (never created in this xdist worker, or removed between check and delete), it's silently ignored.
* Fix OOM and flaky tests in test_process_utils
Replace multiprocessing.Process with subprocess.Popen running minimal
inline scripts. multiprocessing.Process uses fork(), which duplicates
the entire xdist worker memory. At 95% test completion the worker has
accumulated hundreds of MBs; forking it triggers the OOM killer
(exit code 137) on Python 3.14.
subprocess.Popen starts a fresh lightweight process (~10MB) without
copying the parent's memory, avoiding the OOM entirely.
Also replace the racy ps -ax process counting in
TestKillChildProcessesByPids with psutil.pid_exists() checks on the
specific PID — the old approach was non-deterministic because unrelated
processes could start/stop between measurements.
* Add prek hook to validate python_version markers for excluded providers
When a provider declares excluded-python-versions in provider.yaml,
every dependency string referencing that provider in pyproject.toml
must carry a matching python_version marker. Missing markers cause
excluded providers to be silently installed as transitive dependencies
(e.g. aiobotocore pulling in amazon on Python 3.14).
The new check-excluded-provider-markers hook reads exclusions from
provider.yaml and validates all dependency strings in pyproject.toml
at commit time, preventing regressions like the one fixed in the
previous commit.
* Update `uv.lock`
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IntelliJ IDEA .iml files could end up in sdist/wheel packages when
building distributions that lack their own .gitignore. Adding *.iml
to every distribution's .gitignore ensures these IDE metadata files
are excluded from built packages.
* Fix all build-system/requires including transitive dependencies
Add a feature to fix all build-system/requires to make build
reproducibility works. The build-system requires are managed
automatically by the `upgrade-important-versions` prek hook.
This should not only provide reproducibility, but also it should
prevent cases where transient dependency upgrade might break
building sdist (for example as it happened with virtualenv 21
breaking hatchling with https://github.com/pypa/hatch/issues/2193
* Fix mypy 'Cannot infer type of lambda' error in upgrade script
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Remove `UV_HTTP_TIMEOUT`
See https://github.com/apache/airflow/discussions/61205.
Airflow is a large, popular project and a reference for other projects. I'd like to remove the historic use of `UV_HTTP_TIMEOUT`, that isn't required today anymore with the changes in uv. This will also benefit airflow, as broken network connections will fail much faster (either triggering a retry quicker, or reporting to the user in less than 4 x 300s = 20min)
I've kept the CLI option in the dev script to not break existing invocations, I can also remove it if you think it's safe
* Remove `--uv-http-timeout` from launchers too
While we have some discussion on-going whether we should use some
shorter, machine-readable friendly versions of licence specification
in our source code headers here [1], the notion is that:
a) PMC can make judgment calls when to include different versions of
the licence
b) This expectation only applies to the code we actually release
in our official releases.
This change makes some judgment call on using much shorter, SPDX
driven licence headers in some specific files:
* markdown files that are intended to be consumed by agents
(AGENTS.md, SKILLS.md, CLAUDE.md and so on)
* all the markdown .github/* files that are clearly meta-data for
GitHub and which we exclude from released sources
We also make sure all those files are excluded from the official
source releases and distribution packages we prepare.
[1] https://lists.apache.org/thread/j1tn63r2lf13v3d1tnnqff8fkcl4nx53
* Add description on how 3rd-party dependency security issues are handled
Our users seem to not have a good idea on how 3rd-party dependencies
are handled and how they should approach it and open issues that are
often closed and we need to explain them what is expected of them -
they have pretty unrealistic expecations that every single CVE in
every single dependency will be addressed when they open an issue.
This description clarifies how handling of 3rd-party dependency
issues should be done and what are responsibilities and expectations
of the users, and what they can expect from the maintainers.
This will help us to direct such users to this process without
spending our time on explaining it over and over again.
Apply suggestions from code review
Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com>
* Update airflow-core/docs/security/vulnerabilities-in-3rd-party-dependencies.rst
Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
* Update airflow-core/docs/security/vulnerabilities-in-3rd-party-dependencies.rst
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
---------
Co-authored-by: Jens Scheffler <95105677+jscheffl@users.noreply.github.com>
Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
github-actions[bot]andJarek Potiuk
Ash Berlin-Taylor
Shahar EpsteinandJarek Potiuk
bujjibabukatta
Arkadiusz Bach
Andrew Chang
Jens Scheffler
Christos Bisias
DaveT1991andCopilot
Jason(Zhe-You) Liu
Wei Lee
Dev-iL
konsti
Srabasti Banerjee
Guan-Ming (Wesley) Chiu
shubhamkotiandShubham Koti