* [main] CI: Upgrade important CI environment
* Ignore pydantic-ai's generic OpenAI-compatible provider in vendor drift checks
pydantic-ai-slim 2.63.0 adds a provider for any OpenAI-compatible endpoint. It needs a service name at construction and infer_model has no prefix for it, so a common.ai connection can never reach it and it is not a vendor the services page should list.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* Require Python 3.11 in all distributions
Every pyproject.toml that declares requires-python now says >= 3.11 and
drops the Python 3.10 trove classifier, the tomli build requirement that
only existed for 3.10 and the per-version dependency markers for 3.10.
Lock files are regenerated.
With 3.10 gone, the tomli fallbacks in scripts, dev tooling and tests go
too, along with the tomli additional_dependencies of the prek hooks and
the blacken-docs py310 target. The tomllib imports that only sat inside
functions because of the fallback move to the top of their modules. The
four dev/registry and scripts/ci/prek modules left out of the earlier
UP017 pass also take their timezone.utc to datetime.UTC fixes here. The
minimum host Python checked by check-min-python-version, the default
Python of run_generate_constraints.py and the provider cut-off date
parsing (fromisoformat, which reads a trailing Z from 3.11) follow the
new baseline.
ruff infers py311 from the new requires-python, which enables the
PEP 646 star-index fix (UP044) applied here and makes two more rules
reportable: UP042 (str, Enum to StrEnum) alters what str() and
format() return for serialized enum values, and ASYNC109 means
dropping the public timeout parameter from async APIs. Both need
per-site judgement, so they go on the ignore list with that rationale.
The datamodel-codegen target moves to 3.11 too, with use-specialized-enum
turned off for the same reason: otherwise the generated Task SDK and
airflowctl enums would switch from (str, Enum) to StrEnum.
* Drop the Python 3.10 provider metadata shim
With provider distributions now requiring Python 3.11, breeze can derive
requires-python and classifiers from the default and allowed versions
again. The patch-exclusion test and the matching example in
MANAGING_PROVIDERS_LIFECYCLE.rst move from 3.10.0 to 3.11.0, the oldest
patch release CI still covers.
* Keep Breeze's locked dependency versions when dropping Python 3.10
Relocking Breeze for the Python 3.11 baseline ran while recently bumped versions were still inside the 4-day exclude-newer window, so it downgraded pyjwt (reverting #73969), prek, boto3/botocore and filelock. Relocking from the previous lock keeps them and only drops the packages Python 3.11 no longer needs.
Generated-by: Claude Opus 5
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
The SQLAlchemy 2.1.0 sdist ships a pyproject.toml with a duplicate normalized extra (mssql-pymssql), which uv refuses to parse, so the constraints jobs fail building it whenever highest resolution selects it.
Generated-by: Claude Opus 5
* [main] CI: Upgrade important CI environment
* Cap azure-datalake-store below 1 in the Microsoft Azure provider
azure-datalake-store 1.x removed the lib.auth API that AzureDataLakeHook
uses, so the floor bump to 1.0.1 in the CI upgrade broke the hook and
every TestAzureDataLakeHook test. Supporting 1.x is tracked in #73795.
Generated-by: Claude Opus 5
* Regenerate provider requirement tables after the floor bumps
The floor bumps changed provider pyproject.toml files without regenerating
the README.rst and docs/index.rst requirement tables derived from them,
so update-providers-build-files and sync-provider-readme failed static checks.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This reverts commit ecfa20322b.
Main is broken, so reverting this for now.
Errors:
```
azure-ai-projects 2.7.0 has requirement openai>=3.0.0, but you have openai 2.54.0.
```
The provider docs index page only listed cross-provider extras (extras
that pull in another Airflow provider). A provider's plain-PyPI optional
extras -- common-ai's anthropic/openai/langchain, amazon's aiobotocore/
s3fs, and so on -- had no rendering path in the docs at all.
Render the already-computed optional-dependencies table in the provider
index template, mirroring what the PyPI README template already does.
The shared index template drives every provider's index.rst, and the
update-providers-build-files check regenerates them all, so this also
regenerates the index for every provider that declares optional extras.
* Filter required providers from cross-provider extras
Generated provider docs should only advertise extras that users can install explicitly. Required provider dependencies are already installed with the base provider package.
* Clarify optional cross-provider dependency docs
The generated section lists extras that users may install explicitly, so its heading should not imply it contains every cross-provider dependency.
* Always render download links in generated provider docs
The "Downloading official packages" section was inside the
`{% if CROSS_PROVIDERS_DEPENDENCIES %}` block, so providers whose
filtered cross-provider list is empty would silently lose their
download links. Move the section out of the conditional so it renders
for every provider regardless of optional extras.
* Preserve unhyphenated cross provider spelling in docs heading
Renaming the section to "Optional cross-provider package dependencies"
also introduced a hyphen that the surrounding docs and the original
heading did not use. Keep the established "cross provider" spelling so
the new "Optional" prefix is the only change visible to readers.
* Regenerate provider docs after cross provider extras filter
Reapply provider index and README templates so each provider's docs
match the filter that hides cross provider dependencies already
installed as required dependencies, and so they pick up the heading
rename and the relocated "Downloading official packages" section.
* Regenerate provider docs picked up after syncing main
After syncing main into the branch, two providers (ibm.mq, informatica)
need their docs regenerated so they reflect both the cross provider
extras filter introduced in this PR and the dependency updates that
landed on main in the meantime.
* Regenerate informatica README after syncing main
The previous regen used the per-provider build-files prek hook, which
runs breeze with `--skip-readme`. Run the full template regen so
informatica's README.rst also reflects the common.compat dependency
that landed on main.
* Re-apply cross provider extras filter to informatica docs
Earlier docs regenerations were run through a stale `breeze` install
that `uv` had cached from before the filter logic was simplified. As a
result, the previous regen reused the older filter that kept any
cross-provider entry also listed under `optional-dependencies`, and
informatica's docs still advertised `common.compat` as an optional
extra even though it is also a required dependency. Regenerate against
a fresh breeze build so the filter actually drops it.
* Remove old testing monkeypatch
* Match cross-provider extras by package name, not substring
* Prepare provider documentation 2026-06-02
* Address review feedback on provider changelogs
- amazon: move S3 transfer-operators fix (#67378, vincbeck) and EcsRunTaskOperator
log-level fix (#67180, jscheffl) from Features to Bug Fixes; beautify the
EksPodOperator entry (#65335).
- google: beautify the idle/auto-stop TTL entry (#65653) — drop conventional-commit prefix.
- openlineage: beautify the ProcessPoolExecutor self-heal entry (#67400).
- edge3: reword the 3.8.0 note — the provider still supports Airflow 3.0+, only the
execute-callback feature (#67679) needs 3.3+ (jscheffl); move the Swagger API docs
entry (#67390) to Doc-only; beautify the team_name clarification (#66718).
- apache/drill: move the flit.sdist housekeeping entry (#65861) to the excluded block
to match the kafka convention (jscheffl).
* Make apache/drill 3.3.3 a doc-only release
The only non-excluded drill change in this wave was the flit.sdist housekeeping
entry, which jscheffl asked to exclude — leaving an empty changelog. Promote the
DAG-to-Dag wording change (#66153) into a Doc-only section so drill 3.3.3 ships as
a legitimate doc-only release instead of an empty one.
* Fold post-prep provider commits into changelogs
Incremental-update pass for commits that landed on main after this wave was prepared:
- amazon: Propogate verify/botocore_config in redshift cluster triggers (#67876) → Features
(mirrors the batch-triggers entry #67508).
- databricks: Lock in workflow depends_on parent-key behavior (#66681) → Bug Fixes.
- edge3: Fix Edge worker fork mode reporting supervisor failures as success (#67887) → Bug Fixes.
- google: Migrate Stackdriver logging config to RemoteLogIO pattern (#66513) → Misc.
* Add explicit [tool.flit.sdist] sections to flit-based pyproject.toml files
Prepares for flit 4 making --no-use-vcs the default
(https://github.com/pypa/flit/pull/782, pypa/flit#673). With explicit
sdist include lists, sdist contents no longer depend on git, which also
fixes the worktree fail-fast case from #65771.
The provider template emits namespace __init__.py lines based on
provider-id depth (e.g. microsoft.azure also includes
airflow/providers/microsoft/__init__.py). The 97 flit-based provider
pyproject.toml files were regenerated from the updated template.
dev/breeze, devel-common, and performance got their own include lists.
The flit_core==3.12.0 pin is unchanged - bumping to allow flit 4 is
left to dependabot once flit 4 ships.
* Add [tool.flit.sdist] section to akeyless provider
jpype1 1.7.0 stopped shipping prebuilt macOS arm64 wheels and tries to
build from source against a JDK on Apple Silicon, which breaks
`uv sync` out of the box on those machines. Exclude jpype1 1.7.0 as a
direct dependency of the jdbc provider, scoped to darwin-arm64 only so
every other platform (including macOS x86_64) still resolves to the
latest wheels. Upstream plans to restore the arm64 wheels in 1.7.1
(https://github.com/jpype-project/jpype/issues/1357), after which the
exclusion can be dropped.
* Load hook metadata from YAML without importing Hook class
* Add hook-name to all provider.yaml connection-types
* Add hook-name to connection types and regenerate get_provider_info.py
* Fix ruff import order in connections.py
* fix: import ProvidersManager at top level per review
* Fix provider connection hook display names
* Add iter_connection_type_hook_ui_metadata for connection UI hook metadata
* Prepare providers release 2026-03-24
* Update versions of dependent providers to the next version
* Review feedback by Elad+Jarek
* Catchup with changes on 2026-03-24
* Review feedback by Elad+Jarek
* Catch changes from upstream
* Catch changes from upstream
* Update python version exclusion to 3.15
* Add 3.14 metadata version classifiers and related constants
* Regenerate Breeze command help screenshots
* Assorted workarounds to fix breeze image building
- constraints are skipped entirely
- greenlet pin updated
* Exclude cassandra
* Exclude amazon
* Exclude google
* CI: Only add pydantic extra to Airflow 2 migration tests
Before this fix there were two separate issues in the migration-test setup for Python 3.14:
1. The migration workflow always passes --airflow-extras pydantic.
2. For Python 3.14, the minimum Airflow version is resolved to 3.2.0 by get_min_airflow_version_for_python.py, and apache-airflow[pydantic]==3.2.0 is not a valid thing to install.
So when constraints installation fails, the fallback path tries to install an invalid spec.
* Disable DB migration tests for python 3.14
* Enforce werkzeug 3.x for python 3.14
* Increase K8s executor test timeout for Python 3.14
Python 3.14 changed the default multiprocessing start method from 'fork' to 'forkserver' on Linux. The forkserver start method is slower because each new process must import modules from scratch rather than copying the parent's address space. This makes `multiprocessing.Manager()` initialization take longer, causing the test to exceed its 10s timeout.
* Adapt LocalExecutor tests for Python 3.14 forkserver default
Python 3.14 changed the default multiprocessing start method from
'fork' to 'forkserver' on Linux. Like 'spawn', 'forkserver' doesn't
share the parent's address space, so mock patches applied in the test
process are invisible to worker subprocesses.
- Skip tests that mock across process boundaries on non-fork methods
- Add test_executor_lazy_worker_spawning to verify that non-fork start
methods defer worker creation and skip gc.freeze
- Make test_multiple_team_executors_isolation and
test_global_executor_without_team_name assert the correct worker
count for each start method instead of assuming pre-spawning
- Remove skip from test_clean_stop_on_signal (works on all methods)
and increase timeout from 5s to 30s for forkserver overhead
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Bump dependencies to versions supporting 3.14
* Fix PROD image build failing on Python 3.14 due to excluded providers
The PROD image build installed all provider wheels regardless of Python
version compatibility. Providers like google and amazon that exclude
Python 3.14 were still passed to pip, causing resolution failures (e.g.
ray has no cp314 wheel on PyPI).
Two fixes:
- get_distribution_specs.py now reads each wheel's Requires-Python
metadata and skips incompatible wheels instead of passing them to pip.
- The requires-python specifier generation used !=3.14 which per PEP 440
only excludes 3.14.0, not 3.14.3. Changed to !=3.14.* wildcard.
* Split core test types into 2 matrix groups to avoid OOM on Python 3.14
Non-DB core tests use xdist which runs all test types in a single pytest
process. With 2059 items across 4 workers, memory accumulates until the
OOM killer strikes at ~86% completion (exit code 137).
Split core test types into 2 groups (API/Always/CLI and
Core/Other/Serialization), similar to how provider tests already use
_split_list with NUMBER_OF_LOW_DEP_SLICES. Each group gets ~1000 items,
well under the ~1770 threshold where OOM occurs.
Update selective_checks test expectations to reflect the 2-group split.
* Gracefully handle an already removed password file in fixture
The old code had a check-then-act race (if `os.path.exists` → `os.remove`), which fails when the file doesn't exist at removal time. `contextlib.suppress(FileNotFoundError)` handles this atomically — if the file is missing (never created in this xdist worker, or removed between check and delete), it's silently ignored.
* Fix OOM and flaky tests in test_process_utils
Replace multiprocessing.Process with subprocess.Popen running minimal
inline scripts. multiprocessing.Process uses fork(), which duplicates
the entire xdist worker memory. At 95% test completion the worker has
accumulated hundreds of MBs; forking it triggers the OOM killer
(exit code 137) on Python 3.14.
subprocess.Popen starts a fresh lightweight process (~10MB) without
copying the parent's memory, avoiding the OOM entirely.
Also replace the racy ps -ax process counting in
TestKillChildProcessesByPids with psutil.pid_exists() checks on the
specific PID — the old approach was non-deterministic because unrelated
processes could start/stop between measurements.
* Add prek hook to validate python_version markers for excluded providers
When a provider declares excluded-python-versions in provider.yaml,
every dependency string referencing that provider in pyproject.toml
must carry a matching python_version marker. Missing markers cause
excluded providers to be silently installed as transitive dependencies
(e.g. aiobotocore pulling in amazon on Python 3.14).
The new check-excluded-provider-markers hook reads exclusions from
provider.yaml and validates all dependency strings in pyproject.toml
at commit time, preventing regressions like the one fixed in the
previous commit.
* Update `uv.lock`
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
IntelliJ IDEA .iml files could end up in sdist/wheel packages when
building distributions that lack their own .gitignore. Adding *.iml
to every distribution's .gitignore ensures these IDE metadata files
are excluded from built packages.
AIP-95 defines provider lifecycle stages for governance. This adds a
`lifecycle` enum field to the provider.yaml JSON schema and sets the
initial lifecycle stage for all 99 providers:
- `production` (97 providers): All established, actively maintained providers
- `incubation` (2 providers): common-ai (state: not-ready, v0.0.1),
informatica (new provider per AIP-95 governance process)
The `lifecycle` field is orthogonal to the existing `state` field:
- `state`: build/release readiness (ready, not-ready, suspended, removed)
- `lifecycle`: AIP-95 governance stage (incubation, production, mature, deprecated)
The `mature` and `deprecated` stages are left for the PMC to assign
during quarterly reviews per the AIP-95 process.
* Prepare providers release 2026-01-13
* Upgrade common-compat in core as well
* Make dev and optional dependencies consistent
* Revert provider bump on core
* Add backcompat for exceptions in providers
AirflowException and some others are now imported from common.compat instead of
from airflow.exception or airflow.sdk.exception paving way for
the eventual use of the providers without core installation
Backcompat for deprecated exceptions in tests are also included.
* fixup! Add backcompat for exceptions in providers
* Fix conflicts and pre-commit
* fixup! Fix conflicts and pre-commit
* Revert AirflowException compat import in kubernetes.exceptions
* Update providers common.compat dependencies
* fixup! Update providers common.compat dependencies
* fixup! fixup! Update providers common.compat dependencies
* remove duplicate use next version
* Remove update next version from all non-affected providers
* fixup! fixup! Remove update next version from all non-affected providers
* Update fab hash
* fix static checks
* fixup! fix static checks
* Updates to release process of providers
There are several updates to the release process in preparation for
rotational release management process:
* documentation has been simplified and unnecessary steps and digressions
are move out of the way
* some commands had been simplified, optional steps have been moved
at the end of the doc
* commented out PRs in release notes are automatically removed
from the Issue created for users
* better guideliens on how to iterate with the issue by removing
unneeded PRs
* gh issue create used instead of manually hand-crafting the URL
is used to trigger issue creation
* github token retrieval is simplified for prepare-issues command
* stale ".. Please review " comments were removed from the changelogs
* We are using now ${RELEASE_DATE} subfolder in SVN to store the providers -
this will allow several parallel releases to be run.
* Update providers/amazon/docs/changelog.rst
Co-authored-by: Bugra Ozturk <bugraoz93@users.noreply.github.com>
---------
Co-authored-by: Bugra Ozturk <bugraoz93@users.noreply.github.com>
When rebasing/working on #58138 I **thought** I deleted those
deeply nested LICENSE files in our providers that were installed
together with other files when packages were installed. But since
the files are now embedded as metadata in the packages and this is
the only requirement from the ASF point of view, we can safely
delete them now.
This is continuation of #58138
According to https://www.apache.org/legal/src-headers.html#faq-binaries
all our released source packages and binaries should have both LICENCE
and NOTICE files - even if we do not have any 3rd-party software embedded.
This PR updates it accross the board and makes sure that all our
distributions contain both LICENCE and NOTICE file and that they
are added as "licence" information to all our distribution packages
when they are built.
This PR also modernizes all our pyproject.toml files with regards
to latest metadata for licenses:
* license-file is an array of glob patterns
* license is s license expression
* the deprecated Trove Classifier is removed
Inspired by #58131
* Bump all dependencies "apache-airflow-providers-common-compat>=1.8.0"
* Prepare release for Oct 2025 wave of providers
* fix test
* update more providers
* fixes
Replace version-specific conditional imports with ``common.compat.sdk` layer.
This standardizes compatibility handling across Airflow 2.x and 3.x.
Changes:
- Added apache-airflow-providers-common-compat dependency to both providers
- Replaced all version_compat imports with common.compat.sdk imports
- Simplified import logic by removing conditional version checks
- Updated version_compat.py files to retain only version constants
We should change it to just "Airflow" - we have some providers with
Airflow 2+3, some with only 3 and in a few months there will be
Airflow 3 only. So we should remove Airflow reference altogether,
min version of airflow is explained in "Requirements" table below
github-actions[bot]andJarek Potiuk
Shahar EpsteinandJarek Potiuk
Ash Berlin-Taylor
Vincent
PoAn Yang
Hussein Awala
Dr Alex Mitreandmitre88
Kaxil Naik
Andrew Chang
Rahul Madan
Aaron Chen
Yuseok Jo
Dev-iLandClaude Opus 4.6
Eason09053360
Kacper Muda
Aditi102005
Ephraim Anierobi
GUAN-HAO HUANG
Elad Kalif
GPK