* Bump inline script metadata to Python 3.11
The prek hooks and dev scripts declare the interpreter they run under
in PEP 723 metadata, and prek picks the hook interpreter from it, so
this has to land before any hook starts using 3.11-only APIs. With
Python 3.10 gone from the project they can also drop the tomli backport from their dependency lists, because
tomllib is in the standard library from 3.11.
* Refresh diagram checksums for the Python 3.11 script headers
Replace deprecated login_or_token parameter with Auth.Token() to eliminate
deprecation warnings when running the airflow-github compare command.
Updates all three Github() instantiations in compare, changelog, and
needs_categorization functions to use the new auth parameter.
- Fix `is_cherrypicked` function to detect PRs with multiple reference numbers
- Differentiate between Closed and Merged status for better PR state visibility
- Add PEP 723 script dependencies for easier execution with 'uv run --no-project'
The cherry-pick detection was failing for PRs that had additional PR numbers
appended during cherry-picking (e.g., '(#53955) (#53964)'). This change improves
release management workflow by providing clearer PR status information.
The lazy consensus decision has been made at the devlist to switch
entirely to `uv` as development tool:
link: https://lists.apache.org/thread/6xxdon9lmjx3xh8zw09xc5k9jxb2n256
This PR implements that decision and removes a lot of baggage connected
to using `pip` additionally to uv to install and sync the environment.
It also introduces more consistency in the way how distribution
packages are used in airflow sources - basicaly switching all internal
distributions to use `pyproject.toml` approach and linking them all
together via `uv`'s workspace feature.
This enables much more streamlined development workflows, where any
part of airflow development is manageable using `uv sync` in the right
distribution - opening the way to moving more of the "sub-worfklows"
from the CI image to local virtualenv environment.
Unfortunately, such change cannot be done incrementally, really, because
any change in the project layout drags with itself a lot of changes
in the test/CI/management scripts, so we have to implement one big
PR covering the move.
This PR is "safe" in terms of the airflow and provider's code - it
does not **really** (except occasional imports and type hint changes
resulting from better isolation of packages) change Airflow code nor
it should not affect any airflow or provider code, because it does
not move any of the folder where airflow or provider's code is modified.
It does move the test code - in a number of "auxiliary" distributions
we have. It also moves the `docs` generation code to `devel-common`
and introduces separate conf.py files for every doc package.
What is still NOT done after that move and will be covered in the
follow-up changes:
* isolating docs-building to have separate configuraiton for docs
building per distribution - allowing to run doc build locally
with it's own conf.py file
* moving some of the tests and checks out from breeze container
image up to the local environment (for example mypy checks) and
likely isolating them per-provider
* Constraints are still generated using `pip freeze` and automatically
managed by our custom scripts in `canary` builds - this will be
replaced later by switching to `uv.lock` mechanism.
* potentially, we could merge `devel-common` and `dev` - to be
considered as a follow-up.
* PROD image is stil build with `pip` by default when using
`PyPI` or distribution packages - but we do not support building
the source image with `pip` - when building from sources, uv
is forced internally to install packages. Currently we have
no plans to change default PROD building to use `uv`.
This is the detailed list of changes implemented in this PR:
* uv is now mandatory to install as pre-requisite in order to
develop airflow. We do not support installing airflow for
development with `pip` - there will be a lot of cases where
it will not work for development - including development
dependencies and installing several distributions together.
* removed meta-package `hatch_build.py' and replacing it with
pre-commit automatically modifying declarative pyproject.toml
* stripped down `hatch_build_airflow_core.py` to only cover custom
git and asset build hooks (and renaming the file to `hatch_build.py`
and moving all airflow dependencies to `pyproject.toml`
* converted "loose" packages in airflow repo into distributions:
* docker-tests
* kubernetes-tests
* helm-tests
* dev (here we do not have `src` subfolder - sources are directly
in the distribution, which is for-now inconsistent with other
distributions).
The names of the `_tests` distribution folders have been renamed to
the `-tests` convention to make sure the imports are always
referring to base of each distribution and are not used from the
content root.
* Each eof the distributions (on top of already existing airflow-core,
task-sdk, devel-common and 90+providers has it's own set of
dependencies, and the top-level meta-package workspace root brings
those distributions together allowing to install them all tegether
with a simple `uv sync --all-packages` command and come up with
consistent set of dependencies that are good for all those
packages (yay!). This is used to build CI image with single
common environment to run the tests (with some quirks due to
constraints use where we have to manually list all distributions
until we switch to `uv.lock` mechanism)
* `doc` code is moved to `devel-common` distribution. The `doc` folder
only keeps README informing where the other doc code is, the
spelling_wordlist.txt and start_docs_server.sh. The documentation is
generated in `generated/generated-docs/` folder which is entirely
.gitignored.
* the documentation is now fully moved to:
* `airflow-core/docs` - documentation for Airflow Core
* `providers/**/docs` - documentation for Providers
* `chart/docs` - documentation for Helm Chart
* `task-sdk/docs` - documentation for Task SDK (new format not yet published)
* `docker-stack-docs` - documentation for Docker Stack'
* `providers-summary-docs` - documentation for provider summary page
* `versions` are not dynamically retrieved from `__init__.py` all
of them are synchronized directly to pyproject.toml files - this
way - except the custom build hook - we have no dynamic components
in our `pyproject.toml` properties.
* references to extras were removed from INSTALL and other places,
the only references to extras remains in the user documentation - we
stop using extras for local development, we switch to using
dependency groups.
* backtracking command was removed from breeze - we did not need it
since we started using `uv`
* internal commands (except constraint generation) have been moved to
`uv` from `pip`
* breeze requires `uv` to be installed and expects to be installed by
`uv tool install -e ./dev/breeze`
* pyproject.tomls are dynamically modified when we add a version
suffix dynamically (`--version-suffix-for-pypi`) - only for the
time of building the versions with updated suffix
* `mypy` checks are now consistently used across all the different
distributions and for consistency (and to fix some of the issues
with namespace packages) rather than using "folder" approach
when running mypy checks, even if we run mypy for whole
distribution, we run check on individual files rather than on
a folder. That adds consistency in execution of mypy heursistics.
Rather than using in-container mypy script all the logic of
selection and parameters passed to mypy are in pre-commit code.
For now we are still using CI image to run mypy because mypy is
very sensitive to version of dependencies installed, we should
be able to switch to running mypy locally once we have the
`uv.lock` mechanism incorporated in our workflows.
* lower bounds for dependencies have been set consistently across
all the distributions. With `uv sync` and dependabot, those
should be generally kept consistently for the future
* the `devel-common` dependencies have been groupped together in
`devel-common` extras - including `basic`, `doc`, `doc-gen`, and
`all` which will make it easier to install them for some OS-es
(basic is used as default set of dependencies to cover most
common set of development dependencies to be used for development)
* generated/provider_dependencies.json are not committed to the
repository any longer. They are .gitignored and geberated
on-the-flight as needed (breeze will generate them automatically
when empty and pre-commit will always regenerate them to be
consistent with provider's pyproject.toml files.
* `chart-utils` have been noved to `helm-tests` from `devel-common`
as they were only used there.
* for k8s tests we are using the `uv` main `.venv` environment
rather than creating our own `.build` environment and we use
`uv sync` to keep it in sync
* Updated `uv` version to 0.6.10
* We are using `uv sync` to perform "upgrade to newer depencies"
in `canary` builds and locally
* leveldb has been turned into "dependency group" and removed from
apache-airflow and apache-airflow-core extras, it is now only
available by google provider's leveldb optional extra to install
with `pip`
This is continuation of the separation of the Airflow codebase into
separate distributions. This one splits airflow into two of them:
* apache-airflow - becomes an empty, meta no-code distribution that
only has dependencies to apache-airflow-core and task-sdk
distributions and it has preinstalled provider distributions
added in standard "wheel" distribution. All "extras" lead
either to "apache-airflow-core" extras or to providers - the
dependencies and optional dependencies are calculated differently
depending on "editable" or "standard" mode - in editable mode,
just provider dependencies are installed for preinstalled providers
in standard mode - those preinstalled providers are dependencies.
* the apache-airflow-core distribution contains all airflow core
sources (previously in apache-airflow) and it has no provider
extras. Thanks to that apache-airflow distribution does not
have any dynamically calculated dependencies.
* the apache-airflow-core distribution hs "hatch_build_airflow_core.py"
build hooks that add custom build target and implement custom
cleanup in order to implement compiling assets as part of the build.
* During the move, the following changes were applied for consistency:
* packages when used in context of distribution packages have been
renamed to "distributions" - including all documentations and
commands in breeze to void confusion with import packages
(see
https://packaging.python.org/en/latest/discussions/distribution-package-vs-import-package/)
* all tests in `airflow-core` follow now the same convention
where tests are in `unit`, `system` and `integration` package.
no extra package has been as second level, because all the
provider tests have "<PROVIDER>" there, so we just have to avoid
naming airflow unit."<PROVIDER>" with the same name as provider.
* all tooling in CI/DEV have been updated to follow the new
structure. We should always build to packages now when we
are building them using `breeze`.
* Cleanup leftovers frmo API Connexion
* Cleanup leftovers frmo API Connexion, wipe all leftovers
* Fix selective checks
* Add missing dependecy of connexion to FAB provider
* Clean another CODEOWNER leftover
* Revert exceptions about project tests missing
This PR moves the documentation of CI of ours to inside Breeze
doc folder and splits the documentation in separate docs / chapters
following similar changes done for Breeze docs #36936 and the
contributing docs #36969.
Following #36936 and the fact that GitHub stopped rendering big .rst
files, we also split CONTRIBUTING.rst into multiplet files. It will be
much easier to follow and it will render in GitHub.
The BREEZE.rst document became enormous - enough for it to stop being
rendered by GitHub. This change splits it into multiple smaller
documents - each focusing on a specific aspect of Breeze, making it
possible by the user to focus on only that aspect that the user
is interested at.
We also add nice index that guides the user to know about all the
aspects of Breeze.
This is a more complete fix to #33411. This is also a follow up on
earlier implementation of #33261 that addressed checking if PRs
are merged. This one applies the same pattern to finding commit
but also improves it by checking if the (#NNNNNN) ends the subject
- so even if the PR is in the same form in the message, it will be
filtered out.
The previous "--reverse" quick fix in #33411 had potential of problem in
case there were releated PRs merged before the original PR (which is
quite posssible when you have a series of PRs referring to each other.
When a PR is referenced by other PRs, our dev tool for getting the correct
commit lists the latest commit when looking for the commmit sha but we should get the oldest.
The `airflow-github` tooling had a check if PR has been already
merged, but the PR did not work well when there was already a PR
cherry-picked that referred the PR in question in their log message.
The check was done via `git log --grep=#PR` essentially and so
it happenes that `--grep` matches not only the first line of
the message but the whole log message. So if the log message
contained reference to the cherry-picked PR it was seen as merged.
This change adds ( ) around and $ to the regexp to match not
only the PR number but also parentheses around.
pygithub now requires to specify if a search is for pull request or issues and not both.
Also, a fix on searching old rc versions after removing the asf svn repo
Currently we had a requirement to use `test_` prefix for modules for all
our tests, however this means that we could have missed some of the
tests from pytest collection when they were placed in a module without
the `test_` prefix. This happened already: see #30311 and #30306
The previous attempts to remove it failed, because cassandra tests
seemed to be incompatible with pytest collection when we allowed all
Python files, also there were a few names that caused the pytest
collection to fail with selecting all Python files for collection.
The move is accompanied by converting pytest configuration to
pyproject.toml.
After long investigation, it turned out that the cause of cassandra
failures was pytest assert rewrite with collided with Cython-related
type_codes.py definition of types.
See https://github.com/datastax/python-driver/pull/1142 for PR
that is aimed to fix it on cassandra side, in the meantime we
are manually patching the type_codes.py file from Cassandra by adding
PYTEST_DONT_REWRITE to its docstring when building the CI image for
testing.
Another error was using run_module method which also suffers from
assert rewriting incompatibilities but this could be fixed by using
run_path instead. https://github.com/pytest-dev/pytest/issues/9007
Yet another assert-rewrite problem was that ``test_airflow_context``
also failed with ``--assert=rewrite`` turned on when python_files
were not filtered. No solution was found on how we can disable
the rewrite (apparently it happens for yaml.py file but adding
PYTEST_DONT_REWRITE to yaml.py and related files did not help,
so we had to extract the test to separate test type)
Also test in docker_tests/kubernetes_tests should be run with working
directory being in their folders in order to apply pyproject.toml from
their directores (without specifying asyncio mode).
The following changes are applied:
* conversion of pytest configuration from pytest.ini to
pyproject.toml (for main project, docker_tests and breeze)
* addedd pyproject.toml to breeze, docker_tests, kubernetes_tests
to reflect they need different configuration for pytest (lack
of pytest-asyncio for example)
* adding automated patching of Cassandra type_codes.py with
PYTEST_DONT_REWRITE docstring marker
* add pyproject.toml to docker context in order to include it in
sources of airflow in the image
* changed working directory for k8s suite of breeze commands to be
kubernetes_tests
* remove pytest.ini from docker compose mounting and automated removal
in entrypoin in case old image is used
* rename few breeze function to not start with "test_" so that they
are not collected as test methods
* remove few test dags from collecton by pytest by name
* replace run_module with run_path usage in EKS test and test_www.
* CI workflow is updated to use docker_tests as working directory
for those tests
* perf dags were moved out of the tests/test_utils dir to dev
directory.
* PlainAsserts test type was added and the ``test_airflow_context``
test is run in this separate test type when running parallel
tests by default.
This is a follow up after #26289 and 26290 to automatically add
__future__.annotations import to all new files (except empty __init__).
Also the isort configureation is moved to pyproject toml to be
consistent across manual and pre-commit runs of isort
This PR converts the Helm tests to use the new Python Breeze.
It has all the features of previous Kind breeze command and more.
All the commands are now grupped under k8s group and they are very
easy to use locally (even easier than the previous version).
The CI part is also converted and simplified - i.e. the upgrade
test is now much faster (only tests one upgrade per job and it
runs withing the original Helm/Kubernetes tests jobs so it will
not have the cluster creation overhead.
Most importantly - this is almost the last step before we can get
rid of the old legacy breeze code and one that we can get rid of
the `./breeze-legacy` script because all functionality from the
old breeze has been moved to the Python version with this change.
This removal allows us also to remove a lot of the common library
bash code that is not used any more anywhere - even in CI.
The only change left is running regular tests in parallel.
Closes: #23085
Renamed files are listed by github to start with `{old-filename} -> {new-filename}`.
Currently, these are treated as core files. Here we search the renamed files if it starts with what should be ignored
The labelling workflow has proven to be far less useful than we
thought and some of the recent changes in selective checks made
it largely obsolete. The committers can still add "full tests needed"
label when they think it is needed and there is no need to label
the PRs automatically for that (or any other reason).
For quite a while this workflow is basically a useless noise.
Rich_click provides out-of-the-box beautifying of the click command
line tools to provide 'rich' help output.
This PR changes our internal tools (including the new Breeze2
in-progress effort to use rich click to make our help nicer.
When building the "testing status" issue, don't include things skipped
on the changelog or doc-only changes.
Also, don't add skipped changelog entries in the changelog.
Shahar Epstein
Jens Scheffler
Kaxil Naik
Hussein Awala
Jarek Potiuk
Ephraim Anierobi
Amogh Desai
Andrey Anshin
Pierre Jeambrun
Daniel Standish
Jed Cunningham
Brent Bovenzi
Kamil Breguła
James Timmins