nprasad2077/airflow
1
0 Fork 0
mirror of https://github.com/apache/airflow.git synced 2026-11-04 07:12:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
2a3ee7ffe25f2751e181638e2964dcdb06e0d6f0
airflow/dev/breeze/doc/08_ci_tasks.rst
T
Jarek PotiukandShahar Epstein 2a3ee7ffe2 Auto-approve cheap CI runs of external contributors' PRs (#75455) 
* Auto-approve cheap CI runs of external contributors' PRs

GitHub holds the workflow runs of fork PRs until a maintainer approves
them. With self-hosted runners (AIP-118) the approval gate becomes part
of the trust model, so it should cover all external contributors, but
approving every push by hand does not scale and makes contributors wait.

Runs whose change cannot touch the CI definition and whose CI is cheap
are safe to start without a maintainer. Expensive or large changes, and
anything that changes how CI or runners are chosen, still get a human
look first, which also keeps the load on the shared public runners down.
The CI-minutes estimate lets maintainers see what a run will cost before
approving it.

Generated-by: Claude Opus 5

* Add inline script metadata to the CI-minutes calibration script

The script uses only the standard library, so with a PEP 723 block it runs
with plain `uv run dev/calibrate_ci_minutes_estimates.py` instead of syncing
the whole Breeze environment. Document that invocation in the script and in
.github/AGENTS.md.

* Keep one PR's API or git error from aborting the approval tick

Each PR is now processed in its own try block. A requests.RequestException
or subprocess.CalledProcessError for one PR no longer stops the remaining
PRs or skips the job summary. The failed PR is recorded with failed=True and
shown as "failed" in the summary, keeping its CI-time estimate when one was
computed and giving a short, single-line form of the error as the reason.
The workflow also tries to add the 'need workflow approval' label to it, so
it is not lost once its runs leave the query window. Nothing is approved on
failure.

approve_workflow_runs exits 1 after writing the summary when any PR failed,
so a persistent failure turns the scheduled run red instead of passing
silently.

* Approve the runs of fork CI pull requests unless they change CI

A pull request whose CI runs in the contributor's fork (AIP-120) still
gets pull_request runs in apache/airflow, and GitHub holds them until a
maintainer approves them. Those runs only execute the fork CI gate, so
judging them by the cost and size of project CI left cheap gate-only
runs waiting for a maintainer for no benefit, and a gated run that never
ran could not be cleaned up by the reconciler.

What still needs a maintainer is a change to the files that define CI,
since it could remove the gate or add a workflow that does run. Using
the reconciler's own list keeps the two decisions consistent, and the
author is told why their runs wait instead of only seeing a label.

Generated-by: Claude Opus 5

* Keep CI-minutes estimates true to the runs they approve

A run selects its tests from the labels the PR had when it started, so
approving it on the current labels could start a full-tests run whose
label was removed while it waited. Calibrating from self-hosted and
all-versions runs, and leaving out the jobs that need no CI image, made
the estimate describe runs other than those it is used to approve.

Generated-by: Claude Opus 5

---------

Co-authored-by: Shahar Epstein <60007259+shahar1@users.noreply.github.com>
2026-10-25 20:14:06 +00:00

227 lines
11 KiB
ReStructuredText
Raw Blame History

.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
CI tasks
========
Breeze has a number of commands that are mostly used in CI environment to perform cleanup.
Detailed description of the CI design can be found in `CI design <ci/README.md>`_.
**The outline for this document in GitHub is available at top-right corner button (with 3-dots and 3 lines).**
Those are all the commands available in the ``ci`` group:
.. image:: ./images/output_ci.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci.svg
:width: 100%
:alt: Breeze ci com
Running ci upgrade
------------------
Airflow maintainers can use the ``breeze ci upgrade`` command to perform automated upgrades of important
CI dependencies. This command updates prek hooks, chart dependencies, and other critical CI environment
components. It can automatically create a pull request with the changes.
The command checks if you are on the correct branch (main or a version test branch like v2-10-test) with a
clean repository. If not, it will offer to reset your repository to the latest state from apache/airflow.
The upgrade also raises the lower bounds of a curated set of fast-moving dependencies (boto3,
botocore, Google Cloud and Azure SDKs). The policy lives in ``[tool.airflow.dependency-floors]`` of the
root ``pyproject.toml``: a floor is raised to the newest release that is at least ``min-age`` (180 days)
old, members of a group share one floor, and packages that are capped anywhere or listed under
``exclude`` are never touched. Bumps that make ``uv lock`` fail - with the highest or the lowest-direct
resolution - are rolled back and listed in the PR description together with everything that was
raised or skipped. The provider ``README.rst`` and ``docs/index.rst`` requirement tables are then
regenerated from the changed ``pyproject.toml`` files. Use ``--no-upgrade-dependency-floors`` to skip
both steps. The design is recorded in
`ADR 0018 <adr/0018-raise-dependency-floors-automatically.md>`_.
These are all available flags of ``upgrade`` command:
.. image:: ./images/output_ci_upgrade.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_upgrade.svg
:width: 100%
:alt: Breeze ci upgrade
Running resource check
----------------------
Breeze requires certain resources to be available - disk, memory, CPU. When you enter Breeze's shell,
the resources are checked and information if there is enough resources is displayed. However you can
manually run resource check any time by ``breeze ci resource-check`` command.
These are all available flags of ``resource-check`` command:
.. image:: ./images/output_ci_resource-check.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_resource-check.svg
:width: 100%
:alt: Breeze ci resource-check
Freeing the space
-----------------
When our CI runs a job, it needs all memory and disk it can have. We have a Breeze command that frees
the memory and disk space used. You can also use it clear space locally but it performs a few operations
that might be a bit invasive - such are removing swap file and complete pruning of docker disk space used.
These are all available flags of ``free-space`` command:
.. image:: ./images/output_ci_free-space.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_free-space.svg
:width: 100%
:alt: Breeze ci free-space
Fixing File/Directory Ownership
-------------------------------
On Linux, there is a problem with propagating ownership of created files (a known Docker problem). The
files and directories created in the container are not owned by the host user (but by the root user in our
case). This may prevent you from switching branches, for example, if files owned by the root user are
created within your sources. In case you are on a Linux host and have some files in your sources created
by the root user, you can fix the ownership of those files by running :
.. code-block::
breeze ci fix-ownership
These are all available flags of ``fix-ownership`` command:
.. image:: ./images/output_ci_fix-ownership.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_fix-ownership.svg
:width: 100%
:alt: Breeze ci fix-ownership
Selective check
---------------
When our CI runs a job, it needs to decide which tests to run, whether to build images and how much the test
should be run on multiple combinations of Python, Kubernetes, Backend versions. In order to optimize time
needed to run the CI Builds. You can also use the tool to test what tests will be run when you provide
a specific commit that Breeze should run the tests on.
The selective-check command will produce the set of ``name=value`` pairs of outputs derived
from the context of the commit/PR to be merged via stderr output.
More details about the algorithm used to pick the right tests and the available outputs can be
found in `Selective Checks <ci/04_selective_checks.md>`_.
These are all available flags of ``selective-check`` command:
.. image:: ./images/output_ci_selective-check.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_selective-check.svg
:width: 100%
:alt: Breeze ci selective-check
Getting workflow information
----------------------------
When our CI runs a job, it might be within one of several workflows. Information about those workflows
is stored in GITHUB_CONTEXT. Rather than using some jq/bash commands, we retrieve the necessary information
(like PR labels, event_type, where the job runs on, job description and convert them into GA outputs.
These are all available flags of ``get-workflow-info`` command:
.. image:: ./images/output_ci_get-workflow-info.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_get-workflow-info.svg
:width: 100%
:alt: Breeze ci get-workflow-info
Milestone Tag Assistant
-----------------------
The bot will only be triggered when a new push event occurs on the ``main`` or ``v3-1-test`` branches. It will leave a comment in the corresponding PR (similar to how the current ``automatic-backport`` bot behaves) and set the milestone if it matches the rules below, and it will skip if the PRs that already have a milestone set.
There are two cases for the current rules:
- **CI-related**: no need to set milestone
- **Patch release**: default to the **latest patch-release milestone**
- has a label like ``backport-to-v3-1-test``
- is merged to ``v3-1-test`` version branch
If it cannot determine which milestone should be added, it also adds a comment to remind the committer who merged the PR to add the corresponding milestone. This automation ensures that all PRs that should be included in the patch release are properly tagged, making the release manager's life easier.
.. image:: ./images/output_ci_set-milestone.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_milestone-tag-assistant.svg
:width: 100%
:alt: Milestone Tag Assistant
Checking your fork CI setup
---------------------------
Pull requests from external contributors can run their CI in the contributor's fork (AIP-120).
``breeze ci audit`` checks, read-only, whether your checkout and fork are set up for that: ``gh``
installed and authenticated, the ``upstream`` remote pointing at ``apache/airflow`` and ``origin`` at your
fork, whether you are a committer, collaborator, member of an exempted organisation, pilot participant or
external contributor, whether the ``Tests (AMD)`` workflow in your fork is enabled (or disabled, for those
whose pull requests run project CI), how far your fork's ``main`` is behind, and the branch you are on.
Each failing check prints the command that fixes it; ``--fix`` runs those commands after asking.
These are all available flags of ``audit`` command:
.. image:: ./images/output_ci_audit.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_audit.svg
:width: 100%
:alt: Breeze ci audit
Reconciling pull requests gated to fork CI
------------------------------------------
Pull requests from external contributors can run their CI in the contributor's fork instead of in
``apache/airflow`` (AIP-120). The ``fork-ci-reconcile`` command, run every 15 minutes by the
``fork-ci-monitor.yml`` workflow, is the only place that changes those pull requests: it labels newly
gated pull requests ``fork ci required``, keeps them in draft until the fork run is green, maintains
one state comment and a ``Fork CI / Tests (AMD)`` commit status linking to the run in the fork, closes
pull requests without fork CI activity for a week, closes gated pull requests that made project CI run
by changing workflows, and writes everything it could not classify to a Slack message. It also cancels
the unfinished workflow runs of pull requests, of any author, that have already merged. With
``--dry-run`` it prints the changes instead of applying them. See
`Fork CI for external contributors <ci/08_fork_ci_for_external_contributors.md>`_ for the design.
These are all available flags of ``fork-ci-reconcile`` command:
.. image:: ./images/output_ci_fork-ci-reconcile.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_fork-ci-reconcile.svg
:width: 100%
:alt: Breeze ci fork-ci-reconcile
Approving workflow runs of external contributors
------------------------------------------------
The ``auto-approve-workflow-runs`` workflow runs ``approve-workflow-runs`` every 30 minutes. It approves
the waiting workflow runs of external contributors' PRs to ``main`` whose CI jobs are estimated at no more
than 310 job-minutes, that change at most 100 lines of source code and that do not touch CI definition files, and labels the other PRs ``need workflow approval``. PRs whose CI runs in
the contributor's fork (AIP-120) only run the fork CI gate in ``apache/airflow``, so their runs are
approved whatever their size unless they change CI definition files; such a PR also gets a comment
explaining that its CI change needs a maintainer. For every PR it also
estimates the CI jobs selective checks will run and how long they take, and writes them to the job
summary. Use ``--report-only`` to see the decisions without approving runs or changing labels. The rules
are described in `CI workflows <ci/05_workflows.md>`_.
These are all available flags of ``approve-workflow-runs`` command:
.. image:: ./images/output_ci_approve-workflow-runs.svg
:target: https://raw.githubusercontent.com/apache/airflow/main/dev/breeze/images/output_ci_approve-workflow-runs.svg
:width: 100%
:alt: Breeze ci approve-workflow-runs
-----
Next step: Follow the `Release management tasks <09_release_management_tasks.rst>`_ guide to learn how
release managers are using Breeze to release various Airflow artifacts.
Reference in new issue View Git Blame Copy Permalink