* [main] CI: Upgrade important CI environment
* Ignore pydantic-ai's generic OpenAI-compatible provider in vendor drift checks
pydantic-ai-slim 2.63.0 adds a provider for any OpenAI-compatible endpoint. It needs a service name at construction and infer_model has no prefix for it, so a common.ai connection can never reach it and it is not a vendor the services page should list.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* [main] CI: Upgrade important CI environment
* Bump pypdf to 6.21.4 to fix high-severity advisories
The CI environment upgrade resolved pypdf 6.21.3, which the dependency
review gate rejects (GHSA-mfr5-77xh-9j7h, GHSA-cggf-277g-3q54,
GHSA-5849-834g-hvw5). The fixed 6.21.4 is still inside the 4-day
exclude-newer window, so add a temporary per-package cutoff for it.
Generated-by: Claude Opus 5
* Support pydantic-ai 2.60+ and opensandbox 1.2 in common.ai provider
pydantic-ai 2.60 records a durable streamed model request as the bare
response rather than a response-plus-events wrapper, so replaying or
tracking a streamed request in AirflowDurability failed to decode it.
The same release answers a tool retry with the call's own ToolReturnPart
(outcome "retried") instead of RetryPromptPart, which broke tests that
told retries from results by part type.
opensandbox 1.2 allows an egress rule with ports and no target. When the
enforced policy did not match the requested one, sorting the allowed
targets for the error message raised TypeError on that None, hiding the
real reason the sandbox was refused.
Both work with the older releases too, which the provider still supports.
Generated-by: Claude Opus 5
* Fix provider tests and type checks after the dependency upgrade
The tool-part predicates in the common.ai tests returned a plain bool, so
mypy could not narrow the message part they matched and the providers
type check failed.
On Python 3.14 the upgrade moves snowflake-connector-python to 5.1 and
snowflake-sqlalchemy to 2.1. connector 5.1's connect() no longer goes
through SnowflakeConnection, so patching that class stopped intercepting
the connection and the test tried a real login. snowflake-sqlalchemy
2.0.0 started percent-encoding "?" in the URL password, which the URI
test pinned to the older, unencoded form. Older Python versions keep the
earlier releases, so both tests have to accept either.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Previously to see the same error in docs build as you would get in CI,
you would have to run both the docs build and the spellchecking build.
The issue was that the spellcheck (using sphinx-contrib's spelling) was
defined as a new builder, meaning you could either build HTML, or you
can check spelling. On CI, this didn't result in any extra wall-clock
time as the jobs were parallelized (but this change does remove a job
from the overall matrix), but locally that was almost always not the
case, so this would mean full build to view the HTML, and spell checking
build to look for errors, resulting in it taking twice as long.
The upstream spellchecker was also slower than it should be mainly
because its importable-module filter calls find_spec on every distinct
identifier-like word before consulting the dictionary. Here it runs only
for words that are still misspelled. Word lists, filters and
`:spelling:ignore:` behave as before.
This also handles errors a lot more nicely (hence all the code deletions):
- They are written natively to a JSON file, not scraped from a stream
- They also show up as GH annotations (Where we can attribute it to a
real file) to make fixing them easier on PRs.
Please note: this is lightly breaking for Airflow contributors as
`breeze build-docs --docs-only` and `--spellcheck-only` are gone --
every build checks spelling now as it is essentially free (on the order
of milliseconds per module).
* Require Python 3.11 in all distributions
Every pyproject.toml that declares requires-python now says >= 3.11 and
drops the Python 3.10 trove classifier, the tomli build requirement that
only existed for 3.10 and the per-version dependency markers for 3.10.
Lock files are regenerated.
With 3.10 gone, the tomli fallbacks in scripts, dev tooling and tests go
too, along with the tomli additional_dependencies of the prek hooks and
the blacken-docs py310 target. The tomllib imports that only sat inside
functions because of the fallback move to the top of their modules. The
four dev/registry and scripts/ci/prek modules left out of the earlier
UP017 pass also take their timezone.utc to datetime.UTC fixes here. The
minimum host Python checked by check-min-python-version, the default
Python of run_generate_constraints.py and the provider cut-off date
parsing (fromisoformat, which reads a trailing Z from 3.11) follow the
new baseline.
ruff infers py311 from the new requires-python, which enables the
PEP 646 star-index fix (UP044) applied here and makes two more rules
reportable: UP042 (str, Enum to StrEnum) alters what str() and
format() return for serialized enum values, and ASYNC109 means
dropping the public timeout parameter from async APIs. Both need
per-site judgement, so they go on the ignore list with that rationale.
The datamodel-codegen target moves to 3.11 too, with use-specialized-enum
turned off for the same reason: otherwise the generated Task SDK and
airflowctl enums would switch from (str, Enum) to StrEnum.
* Drop the Python 3.10 provider metadata shim
With provider distributions now requiring Python 3.11, breeze can derive
requires-python and classifiers from the default and allowed versions
again. The patch-exclusion test and the matching example in
MANAGING_PROVIDERS_LIFECYCLE.rst move from 3.10.0 to 3.11.0, the oldest
patch release CI still covers.
* Keep Breeze's locked dependency versions when dropping Python 3.10
Relocking Breeze for the Python 3.11 baseline ran while recently bumped versions were still inside the 4-day exclude-newer window, so it downgraded pyjwt (reverting #73969), prek, boto3/botocore and filelock. Relocking from the previous lock keeps them and only drops the packages Python 3.11 no longer needs.
Generated-by: Claude Opus 5
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* [main] CI: Upgrade important CI environment
* Cap azure-datalake-store below 1 in the Microsoft Azure provider
azure-datalake-store 1.x removed the lib.auth API that AzureDataLakeHook
uses, so the floor bump to 1.0.1 in the CI upgrade broke the hook and
every TestAzureDataLakeHook test. Supporting 1.x is tracked in #73795.
Generated-by: Claude Opus 5
* Regenerate provider requirement tables after the floor bumps
The floor bumps changed provider pyproject.toml files without regenerating
the README.rst and docs/index.rst requirement tables derived from them,
so update-providers-build-files and sync-provider-readme failed static checks.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This reverts commit ecfa20322b.
Main is broken, so reverting this for now.
Errors:
```
azure-ai-projects 2.7.0 has requirement openai>=3.0.0, but you have openai 2.54.0.
```
Set `exclude-newer = "4 days"` in `[tool.uv]` to only consider package
versions uploaded at least 4 days ago. This provides supply-chain security
by giving the community time to identify and remediate compromised releases.
Also downgrade pyodps requirement for Python >= 3.13 from >=0.12.6 to
>=0.12.5.1 since 0.12.6 was published within the cooldown window.
* Update python version exclusion to 3.15
* Add 3.14 metadata version classifiers and related constants
* Regenerate Breeze command help screenshots
* Assorted workarounds to fix breeze image building
- constraints are skipped entirely
- greenlet pin updated
* Exclude cassandra
* Exclude amazon
* Exclude google
* CI: Only add pydantic extra to Airflow 2 migration tests
Before this fix there were two separate issues in the migration-test setup for Python 3.14:
1. The migration workflow always passes --airflow-extras pydantic.
2. For Python 3.14, the minimum Airflow version is resolved to 3.2.0 by get_min_airflow_version_for_python.py, and apache-airflow[pydantic]==3.2.0 is not a valid thing to install.
So when constraints installation fails, the fallback path tries to install an invalid spec.
* Disable DB migration tests for python 3.14
* Enforce werkzeug 3.x for python 3.14
* Increase K8s executor test timeout for Python 3.14
Python 3.14 changed the default multiprocessing start method from 'fork' to 'forkserver' on Linux. The forkserver start method is slower because each new process must import modules from scratch rather than copying the parent's address space. This makes `multiprocessing.Manager()` initialization take longer, causing the test to exceed its 10s timeout.
* Adapt LocalExecutor tests for Python 3.14 forkserver default
Python 3.14 changed the default multiprocessing start method from
'fork' to 'forkserver' on Linux. Like 'spawn', 'forkserver' doesn't
share the parent's address space, so mock patches applied in the test
process are invisible to worker subprocesses.
- Skip tests that mock across process boundaries on non-fork methods
- Add test_executor_lazy_worker_spawning to verify that non-fork start
methods defer worker creation and skip gc.freeze
- Make test_multiple_team_executors_isolation and
test_global_executor_without_team_name assert the correct worker
count for each start method instead of assuming pre-spawning
- Remove skip from test_clean_stop_on_signal (works on all methods)
and increase timeout from 5s to 30s for forkserver overhead
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Bump dependencies to versions supporting 3.14
* Fix PROD image build failing on Python 3.14 due to excluded providers
The PROD image build installed all provider wheels regardless of Python
version compatibility. Providers like google and amazon that exclude
Python 3.14 were still passed to pip, causing resolution failures (e.g.
ray has no cp314 wheel on PyPI).
Two fixes:
- get_distribution_specs.py now reads each wheel's Requires-Python
metadata and skips incompatible wheels instead of passing them to pip.
- The requires-python specifier generation used !=3.14 which per PEP 440
only excludes 3.14.0, not 3.14.3. Changed to !=3.14.* wildcard.
* Split core test types into 2 matrix groups to avoid OOM on Python 3.14
Non-DB core tests use xdist which runs all test types in a single pytest
process. With 2059 items across 4 workers, memory accumulates until the
OOM killer strikes at ~86% completion (exit code 137).
Split core test types into 2 groups (API/Always/CLI and
Core/Other/Serialization), similar to how provider tests already use
_split_list with NUMBER_OF_LOW_DEP_SLICES. Each group gets ~1000 items,
well under the ~1770 threshold where OOM occurs.
Update selective_checks test expectations to reflect the 2-group split.
* Gracefully handle an already removed password file in fixture
The old code had a check-then-act race (if `os.path.exists` → `os.remove`), which fails when the file doesn't exist at removal time. `contextlib.suppress(FileNotFoundError)` handles this atomically — if the file is missing (never created in this xdist worker, or removed between check and delete), it's silently ignored.
* Fix OOM and flaky tests in test_process_utils
Replace multiprocessing.Process with subprocess.Popen running minimal
inline scripts. multiprocessing.Process uses fork(), which duplicates
the entire xdist worker memory. At 95% test completion the worker has
accumulated hundreds of MBs; forking it triggers the OOM killer
(exit code 137) on Python 3.14.
subprocess.Popen starts a fresh lightweight process (~10MB) without
copying the parent's memory, avoiding the OOM entirely.
Also replace the racy ps -ax process counting in
TestKillChildProcessesByPids with psutil.pid_exists() checks on the
specific PID — the old approach was non-deterministic because unrelated
processes could start/stop between measurements.
* Add prek hook to validate python_version markers for excluded providers
When a provider declares excluded-python-versions in provider.yaml,
every dependency string referencing that provider in pyproject.toml
must carry a matching python_version marker. Missing markers cause
excluded providers to be silently installed as transitive dependencies
(e.g. aiobotocore pulling in amazon on Python 3.14).
The new check-excluded-provider-markers hook reads exclusions from
provider.yaml and validates all dependency strings in pyproject.toml
at commit time, preventing regressions like the one fixed in the
previous commit.
* Update `uv.lock`
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix all build-system/requires including transitive dependencies
Add a feature to fix all build-system/requires to make build
reproducibility works. The build-system requires are managed
automatically by the `upgrade-important-versions` prek hook.
This should not only provide reproducibility, but also it should
prevent cases where transient dependency upgrade might break
building sdist (for example as it happened with virtualenv 21
breaking hatchling with https://github.com/pypa/hatch/issues/2193
* Fix mypy 'Cannot infer type of lambda' error in upgrade script
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Since UV and other installers essentially ignore < limit in the
python-requires, a good practice is to simply exclude current Python
versions that we **know** we are not compatible with rather than
using <.
Python 3.14 is a thing now and people will get it by default as
latest versions, but we do not run our tests with Python 3.14
yet (not until March/April 2026). Providers have all
upper-bind exclusion except those that do not support Python 3.13,
but that's ok because they are actually limited by Airflow they
can be installed on. But for distributions like airflow-ctl,
we should explicitly exclude 3.14 until we support and test it.
This PR synchronizes all python-requires in non-provider
distribution to consistently exclude Python 3.14 with !=.
According to https://www.apache.org/legal/src-headers.html#faq-binaries
all our released source packages and binaries should have both LICENCE
and NOTICE files - even if we do not have any 3rd-party software embedded.
This PR updates it accross the board and makes sure that all our
distributions contain both LICENCE and NOTICE file and that they
are added as "licence" information to all our distribution packages
when they are built.
This PR also modernizes all our pyproject.toml files with regards
to latest metadata for licenses:
* license-file is an array of glob patterns
* license is s license expression
* the deprecated Trove Classifier is removed
Inspired by #58131
Moved the prod build to also use the src built python
as done by CI. This let's us iterate faster on our
python versions without needing to wait for the community
image of python.
This involves a number of changes:
* 3.0.5 -> 3.0.6 airflow version change
* use released official packages rather than git repo to install Python
* added wget (it was added in the original image to pull python packages so
added for compatibility) added those flags to python build (same as in the
* original build)
* --with-ensurepip --build="$gnuArch"
* --enable-loadable-sqlite-extension
* --enable-option-checking=fatal
* --enable-shared
* --with-lto
* added cleanup of apt after installing packages
* added removal of .pyc/.test etc. files (saves 350 MB)
* added relinking of symbolic links from /usr/python/bin to /usr/local/bin in
the "main" image as well as in the build image.
* we do not need AIRFLOW_SETUPTOOLS_VERSION any more - this was only added to
upgrade setuptools, because the Python official image had a very old
setuptools version.
* checked all "customize" scripts and make them "work"
* Updated the 'version upgrade" script to upgrade AIRFLOW_PYTHON_VERSION everywhere
* Updated Changelog and documentation to update new ways of building images
* removed installation with GitHub URL (it won't work easily after splitting
to multiple packages - not easily at least) and it's not needed
* all python, pip and similar links are created in /usr/python/bin
* /usr/python/bin is always first in the PATH - before /usr/local/bin
* added changelog entry explaining that Python's installation home has been
moved to /usr/python/ from /usr/local
* removal of installed editable distributions in breeze happens now first and
THEN we install when --use-airflow-version is used.
* LD_LIBRARY_PATH was not set so the shared python libraries could not be
loaded when venv was created
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* Breeze changes for Python 3.13
* Pyproject.toml provider changes for Python 3.13
* README.rst changes for providers for Python 3.13
* Add Python 3.13 support for Airflow.
Added Python 3.13 support across the codebase, including:
* CI/CD workflow updates to handle Python 3.13.
* Dockerfile and environment variable adjustments for compatibility.
* Documentation updates to list Python 3.13 as supported.
* Dependency and code changes for compatibility with new/changed
libraries (e.g., numpy 2.x, greenlet, pendulum).
* Test and plugin manager adjustments for Python 3.13-specific behaviors.
* Removal or skipping of tests and features not compatible with Python 3.13.
* Improved error handling and logging for unsupported plugins/providers
on Python 3.13.
* Updated all provider README.rst files to:
* List Python 3.13 as a supported version.
* Adjust dependency requirements for Python 3.13 (e.g., pandas,
pyarrow, ray, etc.).
* Add or update notes about excluded or conditionally supported
dependencies for Python 3.13.
* Document any known incompatibilities or workarounds for specific
providers.
* Updated all provider pyproject.toml files and automation to:
* Add Python 3.13 to the supported classifiers.
* Adjust requires-python and dependency constraints for Python 3.13.
* Exclude Python 3.13 for providers that are not yet compatible (e.g.,
those depending on Flask AppBuilder).
* Add conditional dependencies for new versions of libraries required
by Python 3.13.
* Updated Breeze and related scripts to:
* Build and test with Python 3.13.
* Update documentation and release scripts to include Python 3.13.
* Ensure all dev tooling and test environments work with Python 3.13.
* Removed remaining FAB related code from core and removed tests for
older FAB provider versions (<1.3.0)
* some of the tests in core still depend on FAB - but we make them
optional - both for parsing (we need to skip some imports) and
execution - because they test a legacy behaviour of embedding
Flask plugins that is still supported in core.
* k8s tests now run also with SimpleAuthManager and SimpleAuthManager
is used automatically when runnin tests in Python 3.13
* migration tests had to be excluded fo Python 3.13 because we cannot
migrate down on Python 3.13 without FAB tables created. The tests
will be brought back when FAB is supported or when we release 3.1
and start working on 3.2, we should be able to migrate down to 3.1
* Weaviate tests have been updated to handle case where weaviate-client
is > 2.10.0 - because it started to use httpx instead of requests,
and Python 3.13 actually unblocks using older version of weaviate
(it was previously blocked indirectly via Flask dependencies and
grpcio).
* Added doc change in our rules on what is the "default" Python version
for airflow images - handling the case where we cannot use "newest"
Python version as default where not all "regular" image providers are
supported in the latest version
* Connection is now flushed in `configure_git_connection_for_dag_bundle`
just before `clear_db_connections` - in order to handle Python 3.13
case where FAB app is not initialized by other fixtures - performing
implicit flush along the way
* Cleaning db connections was added to test_dag_run and test_dags test
in fast_api as the test implicitly rely on having connections
cleared (no test git connections added) - this will make the tests
side-effect free.
* Conditionally skipping some serialization tests involving yandex
and ray on Python 3.13 until they support Python 3.13
Follow up after #52967 -> from later discussions it turned out that
it's not really the ~= that is wrong and ambiguous, but that just
upper-binding of Python version is generally considered as a bad
idea - and it's not Astral's view but it's general consensus that
upper-binding of "python-requires" is bad. Since ~= implies
upper-binding, simply replacing it with >= is likely the best option
we can choose.
Building documentation had always been pretty slow and required
using `breeze build-docs` command. This is becuase of the depdendencies
that were needed to be installed, and non-standard setup of the
providers that were not really separated from airflow code.
With the recent distribution separation implemented in airflow code
and isolation implemented in airlfow and all its providers and thanks
to uv workspace that allows to automatically link multiple distributions
in airflow repository - including common development libraries that
are linked between different distributions in the same workspace,
we can now enable documentation building in a very easy way
(using simple sphinx-build commands, as well as enable some of more
advanced features such as sphinx-autobuild to improve iteration
speed when adding and updating documentation.
This PR reworks and repurposes ``build-docs`` command that has
been used inside the container image to be also usable from
regular uv-run local environment. The script has been modernized
and some old and unused code has been removed:
* lint_checks were not really checking anything and those checks
are covered by provider's template generation
* doc and spelling errors were repeated (and unused) in breeze
* you can use short package names to run teh doc-build script
* the script warns if it is used without ``--group docs``
* the ``docs`` and ``gen-docs`` extras and dependency groups have now
plural form to follow the ``docs`` folder as well
* rich-click is used for nicer `--help`` output and arg handing
The contributing docs have been updatted with dedicated docs building
page, which explains the various ways you can build docs, set
prerequisites and iterate quickly on the documentation. It also
contains explanation about some typical errors you might encounter
when running Sphinx build.
Various spelling issues have been solved (the new way of
running the docs revealed some spelling errors).
github-actions[bot]andJarek Potiuk
Ash Berlin-Taylor
Shahar EpsteinandJarek Potiuk
Jens Scheffler
Jason(Zhe-You) Liu
Wei Lee
Dev-iLandClaude Opus 4.6
Aritra BasuandJarek Potiuk