* [main] CI: Upgrade important CI environment
* Ignore pydantic-ai's generic OpenAI-compatible provider in vendor drift checks
pydantic-ai-slim 2.63.0 adds a provider for any OpenAI-compatible endpoint. It needs a service name at construction and infer_model has no prefix for it, so a common.ai connection can never reach it and it is not a vendor the services page should list.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* [main] CI: Upgrade important CI environment
* Bump pypdf to 6.21.4 to fix high-severity advisories
The CI environment upgrade resolved pypdf 6.21.3, which the dependency
review gate rejects (GHSA-mfr5-77xh-9j7h, GHSA-cggf-277g-3q54,
GHSA-5849-834g-hvw5). The fixed 6.21.4 is still inside the 4-day
exclude-newer window, so add a temporary per-package cutoff for it.
Generated-by: Claude Opus 5
* Support pydantic-ai 2.60+ and opensandbox 1.2 in common.ai provider
pydantic-ai 2.60 records a durable streamed model request as the bare
response rather than a response-plus-events wrapper, so replaying or
tracking a streamed request in AirflowDurability failed to decode it.
The same release answers a tool retry with the call's own ToolReturnPart
(outcome "retried") instead of RetryPromptPart, which broke tests that
told retries from results by part type.
opensandbox 1.2 allows an egress rule with ports and no target. When the
enforced policy did not match the requested one, sorting the allowed
targets for the error message raised TypeError on that None, hiding the
real reason the sandbox was refused.
Both work with the older releases too, which the provider still supports.
Generated-by: Claude Opus 5
* Fix provider tests and type checks after the dependency upgrade
The tool-part predicates in the common.ai tests returned a plain bool, so
mypy could not narrow the message part they matched and the providers
type check failed.
On Python 3.14 the upgrade moves snowflake-connector-python to 5.1 and
snowflake-sqlalchemy to 2.1. connector 5.1's connect() no longer goes
through SnowflakeConnection, so patching that class stopped intercepting
the connection and the test tried a real login. snowflake-sqlalchemy
2.0.0 started percent-encoding "?" in the URL password, which the URI
test pinned to the older, unencoded form. Older Python versions keep the
earlier releases, so both tests have to accept either.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
certifi 2026.10.19 removes the emSign ECC Root CA - G3 root after the eMudhra CA compromise. It is inside the 4-day exclude-newer window, so add a temporary per-package cutoff to both [tool.uv.exclude-newer-package] and [tool.uv.pip.exclude-newer-package], following #75296, and relock certifi.
Bypass the PyPI cooldown window for package managers (pip and uv) so that upgrade automation can immediately pick up the latest versions of build tools without waiting for the cooldown period to expire, while regular packages continue to respect the safety cooldown.
CI, the Dockerfiles and breeze already run uv 0.13.1 (CI takes it from uv.lock). A matching floor makes contributors lock and sync with the same uv version as CI, so lock output does not drift with each person's local uv.
Generated-by: Claude Opus 5
* [main] CI: Upgrade important CI environment
* Fix "prek run update-providers-build-files --all-files"
* Fix CI breakages from the dependency upgrade
- snowflake: snowflake-sqlalchemy (up to 1.11.1) cannot be imported on
SQLAlchemy 2.1, which made ORMSelectCompileState private. Cap
sqlalchemy<2.1 in the provider; the workspace lock resolves back to
SQLAlchemy 2.0.54. Removing the cap is tracked in #75118.
- airflow-core: type the dag_display_name and duration hybrid expressions
against the class (type[...] -> ColumnElement) and fix _get_dialect's
return type, so mypy holds on SQLAlchemy 2.0 and 2.1.
- common.ai: declare the OpenAI Decisions and System One vendors added in
pydantic-ai 2.53/2.55; reject sqlglot >= 30.22's exp.ListStage
(Snowflake LIST/LS @stage) on the allow_writes path; adapt tests to
pydantic-ai's streamed Anthropic requests and new request counting, and
to sqlglot's MySQL "--+" tokenization.
Generated-by: Claude Opus 5
* Type Backfill.duration expression against the class
#75078 dropped the type: ignores in this expression, which only held
while mypy did not check the body; typing cls as type[Backfill] keeps it
clean on SQLAlchemy 2.0 and 2.1, as dag_display_name and
DagRun.duration already are. Also annotate the Anthropic stream fixture
so mypy can index its events.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Wei Lee <dev@wei-lee.me>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
The temporary per-package exclude-newer cutoffs for typesafe-sdk
(2026-09-19) and boat-sdk (2026-10-11) were only meant to let their
first required releases resolve inside the 4-day window. Both dates
have passed, so the cutoffs now freeze CI on typesafe-sdk 0.7.0 and
boat-sdk 1.9.3 while users installing the common.ai extras get newer
releases. Drop them from both blocks and relock the two packages.
The root apache-airflow package has no files of its own, but hatchling's
editable build still walks the entire checkout because .gitignore patterns
only filter files, not directory descent. That walk crosses ts-sdk/node_modules
(tens of thousands of pnpm symlink directories) while the ts-sdk prek hooks run
pnpm install in parallel, so any symlink swapped mid-walk raises
FileNotFoundError and fails hooks that uv sync the workspace, such as
lint-helm-chart.
Pruning .gitignore'd directories during the walk keeps hatchling out of
node_modules entirely; the produced editable wheel is unchanged.
* Require Python 3.11 in all distributions
Every pyproject.toml that declares requires-python now says >= 3.11 and
drops the Python 3.10 trove classifier, the tomli build requirement that
only existed for 3.10 and the per-version dependency markers for 3.10.
Lock files are regenerated.
With 3.10 gone, the tomli fallbacks in scripts, dev tooling and tests go
too, along with the tomli additional_dependencies of the prek hooks and
the blacken-docs py310 target. The tomllib imports that only sat inside
functions because of the fallback move to the top of their modules. The
four dev/registry and scripts/ci/prek modules left out of the earlier
UP017 pass also take their timezone.utc to datetime.UTC fixes here. The
minimum host Python checked by check-min-python-version, the default
Python of run_generate_constraints.py and the provider cut-off date
parsing (fromisoformat, which reads a trailing Z from 3.11) follow the
new baseline.
ruff infers py311 from the new requires-python, which enables the
PEP 646 star-index fix (UP044) applied here and makes two more rules
reportable: UP042 (str, Enum to StrEnum) alters what str() and
format() return for serialized enum values, and ASYNC109 means
dropping the public timeout parameter from async APIs. Both need
per-site judgement, so they go on the ignore list with that rationale.
The datamodel-codegen target moves to 3.11 too, with use-specialized-enum
turned off for the same reason: otherwise the generated Task SDK and
airflowctl enums would switch from (str, Enum) to StrEnum.
* Drop the Python 3.10 provider metadata shim
With provider distributions now requiring Python 3.11, breeze can derive
requires-python and classifiers from the default and allowed versions
again. The patch-exclusion test and the matching example in
MANAGING_PROVIDERS_LIFECYCLE.rst move from 3.10.0 to 3.11.0, the oldest
patch release CI still covers.
* Keep Breeze's locked dependency versions when dropping Python 3.10
Relocking Breeze for the Python 3.11 baseline ran while recently bumped versions were still inside the 4-day exclude-newer window, so it downgraded pyjwt (reverting #73969), prek, boto3/botocore and filelock. Relocking from the previous lock keeps them and only drops the packages Python 3.11 no longer needs.
Generated-by: Claude Opus 5
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
apache-airflow-providers-duckdb 0.1.0 passed the 2026-09-22 vote but
could not be uploaded to PyPI: an earlier project with the same name had
uploaded 0.1.0 and 0.1.1 in 2025 and deleted them, and PyPI never lets a
deleted filename be reused. 0.2.0 is the first minor version after the
deleted ones. The changelog warning keeps users from mistaking those
deleted versions for community releases.
Generated-by: Claude Code (Opus 5.5)
* [main] CI: Upgrade important CI environment
* Cap azure-datalake-store below 1 in the Microsoft Azure provider
azure-datalake-store 1.x removed the lib.auth API that AzureDataLakeHook
uses, so the floor bump to 1.0.1 in the CI upgrade broke the hook and
every TestAzureDataLakeHook test. Supporting 1.x is tracked in #73795.
Generated-by: Claude Opus 5
* Regenerate provider requirement tables after the floor bumps
The floor bumps changed provider pyproject.toml files without regenerating
the README.rst and docs/index.rst requirement tables derived from them,
so update-providers-build-files and sync-provider-readme failed static checks.
Generated-by: Claude Opus 5
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
* Mark the IBM MQ and IBM Db2 providers as ready for release
Both providers were merged with `state: not-ready` (IBM MQ in #62790, IBM
Db2 in #67532) so that they would not be picked up by a release wave before
their first review cycle. Release tooling filters on `state` only, so they
have stayed unreleased ever since: `lifecycle: incubation` never blocked a
release for the other incubating providers (DuckDB, ClickHouse, Vespa,
Anthropic), which all shipped their 0.1.0 in the wave after they were merged.
Flipping the state lets the next provider release wave publish 0.1.0 of both,
which is the precondition for either provider ever accumulating the usage and
issue history that graduation from incubation requires. Both re-enter their
own extra and the `all` extra of the meta-package, guarded by the ARM
platform markers each provider already declares.
* Regenerate breeze command images for the newly ready providers
The release-management commands list the ready providers as choices in
their help output, so marking IBM MQ and IBM Db2 as ready changed five of
the rendered screenshots and their hashes.
* Add dependency floor policy configuration
Generated-by: Claude Opus 5
* Find curated dependency requirements and held-back packages
Generated-by: Claude Opus 5
* Select dependency floor targets outside the cooldown
Generated-by: Claude Opus 5
* Rewrite dependency floors in pyproject files
Generated-by: Claude Opus 5
* Roll back dependency floor bumps that break resolution
Generated-by: Claude Opus 5
* Add upgrade-dependency-floors manual hook
Generated-by: Claude Opus 5
* Raise curated dependency floors in breeze ci upgrade
Generated-by: Claude Opus 5
* Document automatic dependency floor bumps
Generated-by: Claude Opus 5
* Harden dependency floor bumps after review
Floors of workspace members with their own uv.lock (dev/breeze) are left
alone, since the root resolve check does not cover that lock. A rollback
restores only the occurrences a bump changed, a bump applies to all its
files or none, malformed PyPI data skips only the affected package, an
aborted run still leaves a note in the PR body, and an already-open
upgrade PR gets the body of the latest run.
Generated-by: Claude Opus 5
* Address remaining review notes on dependency floor bumps
Maintainers read the upgrade PR to see what moved, so the report now
names the files behind each raised floor. Caps set only in the root
[tool.uv] tables are honoured, an unreachable PyPI no longer stalls the
run for every curated package, and the report's temporary directory is
cleaned up.
Generated-by: Claude Opus 5
The openai>=2.47.0,<3 override replaces every package's openai
requirement, so the resolver no longer rejects azure-ai-projects 2.5+,
which requires openai>=3.0.0 unconditionally. A lock refresh then moved
it to 2.7.0 and the CI image build failed in pip check. Cap it at its
last openai 2.x release, which is what the resolver picks without the
override.
This reverts commit ecfa20322b.
Main is broken, so reverting this for now.
Errors:
```
azure-ai-projects 2.7.0 has requirement openai>=3.0.0, but you have openai 2.54.0.
```
pydantic-ai-slim below 2.33 cannot talk to anthropic 1.x: it hands the SDK an
httpx v1 client that anthropic rejects, and it passes temperature as a
messages.create() kwarg that anthropic 1.x removed. The workspace lock was stuck
on 2.31.1 because pydantic-ai-slim[openai]>=2.33 declares openai>=3 while
llama-index-llms-openai and litellm (via the Google provider's aiplatform
evaluation extra) cap openai below 3.
Raise the provider's floor to 2.33.0 and add a workspace uv override that holds
openai on 2.x, the pairing constraints-main already ships and the one the CI
image's pip check accepts. The lock moves only pydantic-ai-slim and
pydantic-graph to 2.44.0. Published wheels are unchanged apart from the floor.
* Raise common.ai openai and llama-index floors to versions that work at the lowest resolution
With the openai override in place, the lowest-direct-dependency run installs the
provider's own floors instead of the ones upstream caps used to force. openai 2.45
and 2.46 reject the httpx2 client pydantic-ai-slim>=2.33 hands them; 2.47.0 is the
first that accepts it. llama-index-llms-openai 0.6.0 declares llama-index-core>=0.13
but imports a type core only gained in 0.14; 0.6.8 is the first release whose
metadata matches, and it needs core>=0.14.5. The override's lower bound moves to
2.47.0 too, since it replaces the provider's floor during that resolution.
A classifier model answers typed questions -- pick one of these options, a
bounded score, a yes/no probability -- and refuses anything that would need
prose. pydantic-ai 2.45.0 added ``TypeSafeModel`` for TypeSafe's Jev, and its
provider takes exactly the ``api_key`` and ``base_url`` that
``PydanticAIHook`` already passes, so no provider code changes: setting
``model_id`` or the connection's Model field is the whole integration.
Adds the opt-in ``typesafe`` extra, a guide covering when a classifier model is
the right choice and where it fits among the existing operators, and an example
Dag with a branch and a confidence-gated classification.
The extra pulls ``typesafe-sdk``, which is pre-1.0, so it stays opt-in rather
than a base dependency. The temporary per-package ``exclude-newer`` pins are
needed because 2.45.0 was published inside the rolling four-day window.
The provider exposes a DuckDBHook extending DbApiHook, so it works with SQLExecuteQueryOperator and the rest of the common-sql ecosystem, plus a DuckDBExecuteQueryOperator and a duckdb connection type. The connection is optional: with none configured the hook opens an in-memory database.
* [main] CI: Upgrade important CI environment
* Keep workspace-relative action references in CI workflows
zizmor 1.30.0 adds the self-repository audit, which wants every in-repo
`uses: ./...` rewritten to GitHub's `uses: $/...` form. Converting the
workflows proved that two things outside this repository's control stop the
new syntax from working here.
`$/` makes the runner fetch apache/airflow as an action repository, and that
download cannot materialise the symlinks this repository commits: it aborts on
`.claude/skills/airflow-translations`, whose target is not tracked, before any
job starts. Separately, ASF infrastructure's allowlist check does not
recognise the syntax and reports every `$/...` reference as an action missing
from the allowlist.
Nor does the audit's threat model reach us in the meantime -- it guards a
privileged workflow against loading a local action out of a checkout untrusted
code could have altered first, and no Airflow workflow runs on
`pull_request_target` or `workflow_run`.
Generated-by: Claude Code (Opus 5)
Claude-Session: https://claude.ai/code/session_01EbprJE5EYQ2Ng7DwAyQdgB
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
B023 catches the late-binding closure-over-loop-variable footgun where
a function defined inside a loop captures the loop variable by
reference, so every function in the resulting list sees the same
(final) value — a classic silent-bug source in Python
(https://docs.python.org/3/faq/programming.html#why-do-lambdas-defined-in-a-loop-with-different-values-all-return-the-same-result).
The one user-visible fix is in providers/standard/.../triggers/file.py,
where the FileTrigger's os.walk lambda was dispatched to a worker
thread via anyio.to_thread.run_sync while the outer glob iteration
could advance, potentially walking the wrong path.
The rest are pre-existing latent-bug or false-positive sites in a
migration script, breeze translation helpers, sphinx extensions, a
system-test example DAG, the SMTP OAuth2 auth callback, secrets_masker
subclass compat shim, and two provider unit tests — fixed by binding
the loop-derived variable as a default argument on the inner function
or lambda.
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
The extra was dropped and its contents inlined because it capped litellm below
the versions carrying the Python 3.14 security fixes. google-cloud-aiplatform
1.164.0 lifts that cap, so the workaround has nothing left to work around, and
carrying a copy of somebody else's extra only lets the two drift apart.
The floor moves to 1.164.0 because every earlier release still ships the capped
extra.
Fixed upstream in: https://github.com/googleapis/python-aiplatform/issues/7057
A not-ready provider has never been published, so naming it in an extra makes
that extra impossible to satisfy - there is no version on PyPI to resolve to.
It broke constraints generation for 3.3.1rc1, where apache-airflow[all] could
not resolve because it required apache-airflow-providers-ibm-mq.
Only the extras change. The source tree still holds those providers, so they
stay in the workspace and under mypy - which is how CI installs and checks them.
* Clarify ``logging_config_class`` contract and document REMOTE_TASK_LOG
``[logging] logging_config_class`` is documented as a "Logging class" but
actually resolves to a ``logging.config.dictConfig`` dict, and the
``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` side channel that powers
remote log read-back was undocumented. Custom configs silently lost UI log
read-back as a result.
- Document the real contract for ``logging_config_class`` (dict, not class)
and the ``REMOTE_TASK_LOG`` / ``DEFAULT_REMOTE_CONN_ID`` module-level
attributes in the config option help, ``advanced-logging-configuration.rst``,
and the ``discover_remote_log_handler`` docstring.
- Add a startup ``WARNING`` when ``remote_logging`` is on but the user's
logging module is missing ``REMOTE_TASK_LOG``, emitted from
``configure_logging`` after ``dictConfig`` runs so it sees the final state.
* Fix CI error
* CI: Fix pyproject.toml
* Fix pyproject.toml
* Drop LOGGING_CONFIG dict from remote logging docs section
Document only REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID in the new remote logging section; do not show users a LOGGING_CONFIG dict to build.
* Clarify user-defined logging config detection and simplify warning tests
An empty ``logging_config_class`` falls back to the default, so treating
it as user-defined under the old ``user_defined`` name was ambiguous per
review feedback. Rename it to state what it actually checks and document
why the empty-path case is excluded. Also collapse the near-duplicate
``TestWarnIfMissingRemoteTaskLog`` tests into one parametrized test using
the project's ``conf_vars`` helper for the config override, per review
suggestions on apache/airflow#67104.
* Default remote_task_log to None and clarify empty logging_config_class handling
_ActiveLoggingConfig.remote_task_log had no default, so
_warn_if_missing_remote_task_log() raised AttributeError if it ran
before _load_logging_config() ever populated the class. A short
comment also clarifies that the `or DEFAULT_LOGGING_CONFIG_PATH`
fallback intentionally covers an explicitly empty
`logging_config_class = ""`.
* Resolve missing-REMOTE_TASK_LOG warning via get_remote_task_log()
The check read _ActiveLoggingConfig.remote_task_log directly, which is
only populated once something has triggered resolution (previously the
deprecated Elasticsearch/OpenSearch handler self-registration during
dictConfig). A user with a custom logging_config_class whose remote
logging actually resolves through ProvidersManager dispatch -- with no
ES/OS handler in the mix -- got a false-positive warning because the
cache was still cold at check time. Going through get_remote_task_log()
triggers the real resolution lazily, so the check reflects whether
remote logging is actually available.
* [main] CI: Upgrade important CI environment
* Fix plugin validation type check for mypy 2.3.0
mypy 2.3.0 narrows the `inspect.isclass()` guard to `type[object]`, which does
not carry `validate`, so the four shared distributions that symlink
plugins_manager.py failed static checks. The MRO check immediately above
already establishes the class is an AirflowPlugin by name -- it cannot use
issubclass() because the shared library is reachable under two module paths --
so make that knowledge explicit to the type checker.
* Fix breeze tag listing type check for GitPython 3.1.52
GitPython 3.1.52 types `git.ls_remote` as the full union of git-command
return shapes, so mypy no longer accepts calling `splitlines()` on it and the
four static-check hooks that type-check dev/breeze failed. With plain
arguments the call always returns the command output, so say so.
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
`AgentSkillsToolset` now accepts an optional `exclude_resources` list of glob
patterns, forwarded to `pydantic-ai-skills`' `SkillsToolset`, to keep matched
files out of a skill's discovered resources. It mirrors the existing
`exclude_tools` escape hatch and is resolved per-run on the worker.
pydantic-ai-skills 1.2.0 broadened resource discovery from a fixed
seven-extension allowlist to every readable text file in a skill bundle, so
`.sql`/`.toml`/`.jinja` assets are exposed with no extra config. That also
widens what reaches the model, so `exclude_resources` is the trim knob for
secrets or scratch files (e.g. `["*.env", "secrets/*"]`). It removes files from
resource discovery only, not from `run_skill_script` filesystem access, so pair
it with `exclude_tools={"run_skill_script"}` when files are genuinely sensitive.
The `[skills]` floor moves to `pydantic-ai-skills>=1.2.0` (the version that adds
the parameter). 1.2.0 was published inside the repo's 4-day exclude-newer
window, so a temporary per-package `exclude-newer-package` override lets it
resolve; remove it once the window advances past 2026-07-15.
* Consolidate OpenLineage e2e tests into airflow-e2e-tests
The OpenLineage provider e2e suite was introduced as a standalone
`providers-e2e-tests/` framework with its own docker-compose stack,
harness, Dockerfile, breeze command and CI workflow. Review feedback on
the introducing PR pointed out that `airflow-e2e-tests/` already hosts
deployed-stack suites for providers (Elasticsearch/OpenSearch remote
logging, Kafka event-driven), so a second parallel framework duplicates
setup that already exists.
Fold OpenLineage in as an `openlineage` E2E test mode: it reuses the
shared docker-compose base, the testcontainers conftest, and the
`AirflowClient`, adding only an overlay for the OpenLineage env config
and a small runner that drives the provider's system-test DAGs. The
separate framework, its breeze command, workspace member and dedicated
workflow are removed; the older-Airflow compatibility matrix is dropped
along with them.
* Fix OpenLineage e2e system-tests path resolution
The repo root was computed with an off-by-one parents index, so the
system-test DAGs source resolved outside the repository and the e2e
session failed at startup in CI. Reuse AIRFLOW_ROOT_PATH from the e2e
framework constants instead of re-deriving it.
Generated-by: Claude Code (Fable 5)
* Fix OpenLineage e2e crash on 204 responses from variable deletion
The shared AirflowClient parsed every response body as JSON, but the
variable-deletion cleanup in the OpenLineage e2e harness issues DELETE
requests that return 204 No Content, whose empty body is not valid JSON.
* Restore OpenLineage e2e older-Airflow compat matrix and README
Review feedback on the consolidation asked to keep the compatibility
coverage that ran the OpenLineage suite against older released Airflow
versions with current provider code — it catches core-vs-provider
breakages early, which matters for a provider as tightly coupled to core
as OpenLineage. To keep PR CI cheap, the compat matrix now runs on canary
(scheduled / main) only; PRs still run the PROD-image job. Also restores
the suite's README next to the tests for debugging reference.
* Allow triggering OpenLineage e2e compat matrix via full-tests-needed label
The compat matrix was gated to canary-only, giving no way to exercise it
on a PR for a larger OpenLineage change. Add a dedicated
run-openlineage-e2e-compat-tests selective-check output (canary or the
explicit 'full tests needed' label) so maintainers can opt a PR in, while
keeping it off ordinary OpenLineage PRs. Deliberately not tied to derived
full_tests_needed, matching the run-ui-e2e-tests convention.
Rahul Vats
github-actions[bot]andJarek Potiuk
Kaxil Naik
Ash Berlin-Taylor
Dev-iL
Shahar Epstein
Yossi Eliaz
Haseeb Malik
David Blain
Divyansh Yadav
Niko Oliveira
Deepak kumarandJarek Potiuk
PoAn Yang
Wei Lee
Jason(Zhe-You) Liu
Jake McGrath
GPK