* Depend on httpx2 instead of httpx in the Task SDK
httpx has stalled: fixes that landed in httpx2 (a fork of httpx 0.28.1)
have not shipped upstream, and starlette has already migrated, so FastAPI
following is a matter of time. Moving the Execution API client now keeps
the SDK on a maintained HTTP stack.
Objects do not cross the httpx/httpx2 boundary, so the in-process
Execution API transports move with the client that consumes them; the
rest of airflow-core stays on httpx and migrates separately.
related: https://github.com/apache/airflow/issues/70522
* Add newsfragment for the Task SDK httpx2 migration
* Match the in-process test client transport to the installed Task SDK
Compat tests pair the repo's devel-common helpers with a released Airflow
whose Task SDK is still on httpx. Pinning the helper to httpx2 hands those
clients a transport from the wrong package, and httpx rejects the response
stream it gets back.
* Rename newsfragment to the new PR number
* Pin the Task SDK ast-grep hook to a frozen revision
Every other third-party hook in the repo pins an immutable SHA; a mutable
tag lets the hook change under CI without a diff.
* Remove unneeded newsfragment
* Verify Task SDK HTTP logging and simplify test helpers
Both HTTP stacks can coexist while compatibility jobs use older SDK
distributions. The migration needs regression coverage for its logging
behavior without exposing development-only helpers as public APIs.
* Use httpx2 in the Dag run note client test
* Use httpx2 in the execution API shutdown tests
Tests added on main for in-process shutdown still referenced httpx, which this branch no longer imports in the module.
* Trust the system certificate store in the Task SDK API client
httpx2 verifies against the system trust store via truststore, so keeping certifi for the Execution API client would add a direct dependency only to maintain a second, different set of trusted roots in the same worker.
* Ban httpx imports in the Task SDK with ruff
Ruff already enforces banned imports, so a separate ast-grep hook and its extra pre-commit dependency are unnecessary. extend-banned-api adds to the root banned-api table instead of replacing it.
* Use httpx2 in the server termination acknowledgement test
The test arrived on main after the Task SDK moved to httpx2 and still referenced httpx, which the module no longer imports.
* Test the Task SDK SSL context against a real CA file
Mocking load_verify_locations while also writing a CA file to disk tested neither path convincingly. Keeping the certificate-generation imports inside the fixture limits them to the one place that needs them.
* Note when the in-process httpx resolver can be removed
* Apply suggestion from @ashb
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* Use httpx2 in the supervisor launch error test helper
The helper arrived on main after the Task SDK moved to httpx2 and still referenced httpx, which the module no longer imports.
* Remove the duplicated in-process httpx resolver note
* Use httpx2 in the supervisor execution timeout test
The test arrived on main after the Task SDK moved to httpx2 and still referenced httpx, which the module no longer imports.
* Use httpx2 in the request handler tests
The tests arrived on main after the Task SDK moved to httpx2; the client rejects httpx transports.
---------
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* Require Python 3.11 in all distributions
Every pyproject.toml that declares requires-python now says >= 3.11 and
drops the Python 3.10 trove classifier, the tomli build requirement that
only existed for 3.10 and the per-version dependency markers for 3.10.
Lock files are regenerated.
With 3.10 gone, the tomli fallbacks in scripts, dev tooling and tests go
too, along with the tomli additional_dependencies of the prek hooks and
the blacken-docs py310 target. The tomllib imports that only sat inside
functions because of the fallback move to the top of their modules. The
four dev/registry and scripts/ci/prek modules left out of the earlier
UP017 pass also take their timezone.utc to datetime.UTC fixes here. The
minimum host Python checked by check-min-python-version, the default
Python of run_generate_constraints.py and the provider cut-off date
parsing (fromisoformat, which reads a trailing Z from 3.11) follow the
new baseline.
ruff infers py311 from the new requires-python, which enables the
PEP 646 star-index fix (UP044) applied here and makes two more rules
reportable: UP042 (str, Enum to StrEnum) alters what str() and
format() return for serialized enum values, and ASYNC109 means
dropping the public timeout parameter from async APIs. Both need
per-site judgement, so they go on the ignore list with that rationale.
The datamodel-codegen target moves to 3.11 too, with use-specialized-enum
turned off for the same reason: otherwise the generated Task SDK and
airflowctl enums would switch from (str, Enum) to StrEnum.
* Drop the Python 3.10 provider metadata shim
With provider distributions now requiring Python 3.11, breeze can derive
requires-python and classifiers from the default and allowed versions
again. The patch-exclusion test and the matching example in
MANAGING_PROVIDERS_LIFECYCLE.rst move from 3.10.0 to 3.11.0, the oldest
patch release CI still covers.
* Keep Breeze's locked dependency versions when dropping Python 3.10
Relocking Breeze for the Python 3.11 baseline ran while recently bumped versions were still inside the 4-day exclude-newer window, so it downgraded pyjwt (reverting #73969), prek, boto3/botocore and filelock. Relocking from the previous lock keeps them and only drops the packages Python 3.11 no longer needs.
Generated-by: Claude Opus 5
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
CI no longer runs Python 3.10, so the tomli fallbacks for tomllib, the
typing_extensions imports of Self, Unpack and assert_never, the
re-implementation of asyncio.Runner in the Task SDK and the version
gated test skips no longer have a version to serve.
PR #68840 fixed an issue where exceptions thrown by the FastAPI
lifecycle hook were swallowed. In doing so, it exposed a pre-existing
problem where the lifecycle hook couldn't run when the JWT secret was
not provided.
As the `InProcessExecutionAPI` overrides auth, it doesn't need a JWT
secret, and we certainly don't want to start crashing processes that
previously ran fine as a result of a missing secret that we don't need.
This commit stubs out the registered `JWTValidator` in the FastAPI app
created for the `InProcessExecutionAPI`. This is done in an isolated
services registry to ensure we don't leak this into any real app
instantiations.
Through `a2wsgi.ASGIMiddleware`, the `InProcessExecutionAPI` was leaking
a daemon thread per instance, used by `a2wsgi` to run an asyncio event
loop.
This commit moves management of that background thread into
`InProcessExecutionAPI`, and adds a finalizer so that the thread is
stopped when the execution API instance is garbage collected.
As Cadwyn 6.0.0 is released with compatibility with FastAPI 0.128.0+,
we can now attempt to run the tests with it and see what else needs
to be fixed.
Related: https://github.com/zmievsa/cadwyn/issues/316
The pre-commit is a fantastic tool, and we heavily used it for years,
but generally the tool stagnated and is not showing a sign of adapting
to our needs. For years we tried to convince pre-commit maintainers that
things like autocomplete are necessary - but it met with pretty much
resistance (if not hostility) from the maintainer. Also there was no
chance for them to accept expectations of bigger projects like ours,
where we have a huge monorepo and not only multiple needs but also
different parts of the repo needing different language support (golang,
typescript soon) - and apparenty the maintainer of pre-commit does not
think monorepo is a good thing at all. Similarly - they did not recognize
the raise of `uv` and the only way to use `uv` with pre-commit is to patch
it by installing `pre-comit-uv` that essentialy patches pre-commit with
uv support. This is not really sustainable and the tool lags behind many
of our needs.
Luckily - we have new project in town - prek - which rewrites pre-commit
that is 100% compatible (now), 10x faster (because rust), uses `uv`
natively, supports auto-complete already and they have very friendly
maintainer who is not only supporting us but also very happily works
on improving `prek` to close all the gaps, and plans to implement (with
our support of course and cooperation) monorepo support - that will
allow us to modularise our pre-commits.
This PR switches our pre-commit support to use prek exclusively:
* breeze static checks command is completely removed
* custom auto-complete code in breeze as well
* instructions are updated to setup prek instead of precommit
* CI is updated to run prek instead of pre-commmit
* documentation for static checks is reviewed and new features that
prek enables are added
This change improves the generated code using the OpenAPI schema:
- `anyOf` to `oneOf` so that generated clients don't try _all_ schemas, but
stop after the first
- Fix the header pattern to make it valid against the spec
- Use the function name as the default operation ID in the spec.
On this last part, this is a different implementation (but same effect) to how
we do operation name in the Core API. I will change CoreAPI to use this same
method in a separate PR, but I didn't want to touch both APIs in this PR for
clarity of git history
We'd already removed any meaningfull checks in this endpoint in a previous PR
Although right now we don't have any API clients calling the previous version
since we aren't released yet and could have simply deleted the endpoint
outright, this is a good opportunity to use this as a practice run for API
versioning
---------
Co-authored-by: Sneha Prabhu <snehaprabhu@Snehas-MacBook-Pro.local>
Co-authored-by: Ash Berlin-Taylor <ash@apache.org>
This is continuation of the separation of the Airflow codebase into
separate distributions. This one splits airflow into two of them:
* apache-airflow - becomes an empty, meta no-code distribution that
only has dependencies to apache-airflow-core and task-sdk
distributions and it has preinstalled provider distributions
added in standard "wheel" distribution. All "extras" lead
either to "apache-airflow-core" extras or to providers - the
dependencies and optional dependencies are calculated differently
depending on "editable" or "standard" mode - in editable mode,
just provider dependencies are installed for preinstalled providers
in standard mode - those preinstalled providers are dependencies.
* the apache-airflow-core distribution contains all airflow core
sources (previously in apache-airflow) and it has no provider
extras. Thanks to that apache-airflow distribution does not
have any dynamically calculated dependencies.
* the apache-airflow-core distribution hs "hatch_build_airflow_core.py"
build hooks that add custom build target and implement custom
cleanup in order to implement compiling assets as part of the build.
* During the move, the following changes were applied for consistency:
* packages when used in context of distribution packages have been
renamed to "distributions" - including all documentations and
commands in breeze to void confusion with import packages
(see
https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/)
* all tests in `airflow-core` follow now the same convention
where tests are in `unit`, `system` and `integration` package.
no extra package has been as second level, because all the
provider tests have "<PROVIDER>" there, so we just have to avoid
naming airflow unit."<PROVIDER>" with the same name as provider.
* all tooling in CI/DEV have been updated to follow the new
structure. We should always build to packages now when we
are building them using `breeze`.
Right now we have exactly one version, but using Cadwyn gives us the ability
to make changes to the Execution API almost "at will" and it will
transparently upgrade from older request versions. The tl;dr of it is "apply
migrations to your requests so that you only have to maintain the latest
version".
From it's docs[^1]:
> Cadwyn allows you to support a single version of your code while
> auto-generating the schemas and routes for older versions. You keep API
> versioning encapsulated in small and independent "version change" modules
> while your business logic stays simple and knows nothing about versioning.
This gives us the freedom to change the version of the API but continue to
support "all" old versions. This is a key aspect of proposal in AIP-72. We
don't yet have any new versions -- those will come later.
[^1]: https://docs.cadwyn.dev/
This is the next stage of refactoring of airflow packages, after
moving providers to standalone dstribution and separating devel-common
as a common distribution.
The `task_sdk` has been renamed to `task-sdk` - this way we will
never import anything in task_sdk accidentally starting from content
root. Some changes have been needed to make it works:
* autouse fixture was added to pytest plugin to add `task-sdk/tests`
to PYTHONPATH to make it root import
* all tests were moved to `task_sdk` package inside the tests folder
* all imports for tests are now `from task_sdk`
* common tools for task_sdk has been moved to
`devel-common/src/test_utils/task_sdk.py` in order to allow importing
them before `task-sdk/tests` is added to pythonpath
ZachandAsh Berlin-Taylor
Shahar EpsteinandJarek Potiuk
Nick Stenning
Pierre Jeambrun
Ash Berlin-Taylor
