This is continuation of the separation of the Airflow codebase into
separate distributions. This one splits airflow into two of them:
* apache-airflow - becomes an empty, meta no-code distribution that
only has dependencies to apache-airflow-core and task-sdk
distributions and it has preinstalled provider distributions
added in standard "wheel" distribution. All "extras" lead
either to "apache-airflow-core" extras or to providers - the
dependencies and optional dependencies are calculated differently
depending on "editable" or "standard" mode - in editable mode,
just provider dependencies are installed for preinstalled providers
in standard mode - those preinstalled providers are dependencies.
* the apache-airflow-core distribution contains all airflow core
sources (previously in apache-airflow) and it has no provider
extras. Thanks to that apache-airflow distribution does not
have any dynamically calculated dependencies.
* the apache-airflow-core distribution hs "hatch_build_airflow_core.py"
build hooks that add custom build target and implement custom
cleanup in order to implement compiling assets as part of the build.
* During the move, the following changes were applied for consistency:
* packages when used in context of distribution packages have been
renamed to "distributions" - including all documentations and
commands in breeze to void confusion with import packages
(see
https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/)
* all tests in `airflow-core` follow now the same convention
where tests are in `unit`, `system` and `integration` package.
no extra package has been as second level, because all the
provider tests have "<PROVIDER>" there, so we just have to avoid
naming airflow unit."<PROVIDER>" with the same name as provider.
* all tooling in CI/DEV have been updated to follow the new
structure. We should always build to packages now when we
are building them using `breeze`.
Right now we have exactly one version, but using Cadwyn gives us the ability
to make changes to the Execution API almost "at will" and it will
transparently upgrade from older request versions. The tl;dr of it is "apply
migrations to your requests so that you only have to maintain the latest
version".
From it's docs[^1]:
> Cadwyn allows you to support a single version of your code while
> auto-generating the schemas and routes for older versions. You keep API
> versioning encapsulated in small and independent "version change" modules
> while your business logic stays simple and knows nothing about versioning.
This gives us the freedom to change the version of the API but continue to
support "all" old versions. This is a key aspect of proposal in AIP-72. We
don't yet have any new versions -- those will come later.
[^1]: https://docs.cadwyn.dev/
This is the next stage of refactoring of airflow packages, after
moving providers to standalone dstribution and separating devel-common
as a common distribution.
The `task_sdk` has been renamed to `task-sdk` - this way we will
never import anything in task_sdk accidentally starting from content
root. Some changes have been needed to make it works:
* autouse fixture was added to pytest plugin to add `task-sdk/tests`
to PYTHONPATH to make it root import
* all tests were moved to `task_sdk` package inside the tests folder
* all imports for tests are now `from task_sdk`
* common tools for task_sdk has been moved to
`devel-common/src/test_utils/task_sdk.py` in order to allow importing
them before `task-sdk/tests` is added to pythonpath
Jarek Potiuk
Ash Berlin-Taylor