* Tell agents to stop when breeze cannot run instead of using a host venv
AGENTS.md said never to run pytest on the host but also listed
`uv run --project` commands, and had no guidance for when breeze is
unavailable. A sandboxed agent whose breeze calls were refused by the
container gateway built a host virtualenv with Airflow 2.11.1 instead of
reporting the failure.
Narrow the host exception to the listed `uv run` commands, tell agents
to stop and report when breeze cannot run, and document the breeze
command for testing providers against a released Airflow version
(generated from contributing-docs). Note in the magpie override that the
breeze allows do not make breeze work behind the container gateway.
* Record skill-eval run for the breeze-unavailable AGENTS.md guidance
* Clarify when to add a newsfragment
The contribution guide, AGENTS.md and the PR drafting skill each had
their own newsfragment rule, and they disagreed: the guide encouraged
adding one of six types, AGENTS.md said "major or breaking" but also
asked for one on any user-facing change. The dev@ thread "[DISCUSS]
Newsfragments: when to add one" proposed significant-only fragments
for airflow-core, no fragment for an ordinary bug fix, feature or doc
change, and one rule written once with AGENTS.md pointing at it.
* Tighten the newsfragment criteria after review
task-sdk has no release notes of its own, its changes are listed in
the Airflow release notes, which is why its newsfragments go to
airflow-core. A change that is neither a behaviour change nor needs
migration steps does not need a newsfragment, and listing every
towncrier type invited using the ones the rule excludes.
* Add breeze verify to show CI-required local verification for changed files
The self-review checklist in AGENTS.md (step 7) asks contributors to find
the tests selective-checks would run and run them locally. The only tool
for that is `breeze ci selective-check`, which looks at a single commit
and prints GitHub Actions outputs, so a contributor still has to read the
workflow files to turn those outputs into commands. This adds a
developer-facing, dry-run command that feeds the working-tree diff to the
same SelectiveChecks logic and prints the commands for the default CI
matrix cell, with the rest explicitly left to CI.
* Print the CI reproduction banner on stderr so breeze verify --json stays parseable
In GitHub Actions breeze appends a "how to reproduce locally" banner after
every command. It went to stdout, so any command whose stdout is a machine
contract, like `breeze verify --json`, produced JSON followed by the banner.
The banner is a log aid, so stderr is the right stream for it.
* Let a breeze command opt out of the CI reproduction banner instead of moving it to stderr
Moving the banner to stderr broke the build-info job: `breeze ci
get-workflow-info` and `breeze ci selective-check` write their GitHub
Actions outputs on stderr (`2>> $GITHUB_OUTPUT`), so the banner ended up
in the outputs file. Neither stream is safe for every command. Keep the
banner on stdout and let a command whose stdout is a machine contract,
here `breeze verify --json`, switch it off through the click context.
* List renamed files the way CI does when computing breeze verify changed files
CI feeds selective checks the output of `git diff-tree`, which has no
rename detection and reports a rename as the old path deleted plus the
new path added. `git diff --name-only` detects renames by default and
lists only the new path, so a rule matching the old path would not fire
locally although it fires in CI.
* Only list jobs triggered by the change in breeze verify
Breeze's own unit tests, the shared distribution tests and the translation
check run on every PR regardless of what changed, so listing them on every
run adds rows nobody acts on and pushes the relevant ones down. Now every
row comes from a SelectiveChecks output. The `required` column goes with
them: with the translation check gone it was always true.
* Make breeze verify list what to run by default and keep the CI expansion behind --full
When a change touches CI tooling or dependency files, selective checks
make CI run the whole suite. That is the right call for CI, but an agent
that executes the list literally would run Go, Java and every provider
for a breeze change. The default now applies the same selective-checks
rules without that expansion, notes that CI will run more, and --full
lists the whole default matrix cell.
* Fold long commands in the breeze verify table instead of truncating them
A truncated command cannot be copied, and the prek row with its SKIP list
is longer than most terminals.
* Print breeze verify commands one per line instead of in a table
A table cell is not something you can copy: long commands got truncated,
and folding them only spread one command over several lines. Each command
is now on its own unwrapped line under a comment saying what kind of check
it is and where it runs, so a triple-click selects the whole command.
* Keep the table output in breeze verify
Reverts the one-command-per-line output. The table is easier to read, and
anyone who wants to copy commands has --json.
* List the jobs CI runs on every PR under breeze verify --full
--full means everything CI runs for the default matrix cell, so breeze's
own unit tests and the shared distribution tests belong there even though
no change triggers them. The default list still leaves them out. The
translation check stays out of both: CI runs it with `|| true`.
* Address review: guard hook names, use --from-ref by default, point agents at breeze verify
The manual-stage hook names in the mapping are now checked against
ci-amd.yml, so a renamed hook fails a test instead of going stale. The
default prek row uses --from-ref, the scope AGENTS.md recommends, and
--full keeps CI's --all-files form. Step 7 of the publish-changes skill,
the AGENTS.md commands block and the selective-checks doc now point at
breeze verify. All patches in the CLI tests use autospec.
* Classify breeze verify flags from the workflow gates instead of the run_ prefix
The classification test used to enumerate SelectiveChecks properties named
run_* and assume those were the job gates, so a gate with another name
would slip through. It now scans every `if:` in .github/workflows/ for
the selective-checks outputs CI actually gates on, which also drops the
hand-kept list of run_* outputs no job reads.
* Scan complete multiline workflow conditions
* Keep breeze verify classification focused on run flags
* Regenerate breeze command images and skill-eval hash after rebase
* Leave static checks to prek in breeze verify
prek already picks the hooks to run for the changed files, so breeze
verify no longer lists a prek row, in either mode. The mypy-providers
and migration-round-trip jobs are manual-stage prek hooks too; their
selective-checks flags are now classified in PREK_JOBS and not printed.
The output, the --full help, the breeze docs, AGENTS.md and the
publish-changes skill say static checks are not listed.
Also classify run_agent_framework_tests, added to SelectiveChecks on
main, with the breeze shell commands its CI job runs per framework.
* Compare breeze verify with main on the apache/airflow remote by default
The default --base-ref was the local main branch. In a fork it is often
behind the main the branch has merged (a GitHub "Update branch" or a
conflict fix), and then every merged-in commit counted as a change:
3287 files instead of the 21 the PR changes.
The default is now main on the remote whose URL points at apache/airflow,
preferring upstream, which is the base GitHub compares the PR with.
Without such a remote it falls back to the local main and says so.
The local copy of that remote can be stale too, after "Update branch"
without a fetch. breeze verify now warns when a merge on the branch
brought in commits the base does not have. With --json the warnings go
to stderr so stdout stays parseable.
* Point breeze verify at the prek hooks a default install does not run
Dropping the static-check rows also hid mypy-providers and migration-round-trip, which CI runs but a default prek install never triggers (pre-push and manual stage), so a migration change got no hint until CI failed. The CLI tests also ran real git against the developer's checkout and global config.
Generated-by: Claude Opus 5
---------
Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
Contributors are asked to test their changes in every providers wave, but
finding the right distributions, installing the whole wave together and
building a reproduction for each change is tedious enough that it is often
skipped. Adopting the release-management family, with an override that
automates the mechanical part, lets an agent do that while the contributor
looks at the behaviour.
Generated-by: Claude Opus 5
Reviewers need a signal that a person understood and owns an agent-assisted
change. A short summary in the author's own words, kept separate from the
agent's description, gives them that - but only if the agent never writes it.
Generated-by: Claude Opus 5
The default state store backend class is named MetastoreBackend. Readers
following the docs landed on a name that does not exist, and the provider
docs cross-reference to it could not resolve.
Breeze was installed by resolving dev/breeze/pyproject.toml against the package
index on every fresh environment — `uvx --from ./dev/breeze` in the shim, `uv tool
install --editable ./dev/breeze` in CI. Neither reads dev/breeze/uv.lock, and
neither applies the `[tool.uv] exclude-newer` buffer declared beside it, which
governs project operations only. The committed lock recorded nothing, so an
upstream release could change what breeze runs with no commit in this repository.
click 8.5.0 showed the cost. It added a `help` field to the argument info dict
breeze hashes to detect command drift, so within the hour of its release every CI
job resolved it, every command taking a positional argument hashed differently
from its committed value, and static checks went red on every open PR regardless
of what it touched. The lock said click 8.4.2 throughout.
The command-output files move back to the versions the lock produces; the next
lock upgrade regenerates them together with the dependency bump that causes them.
The snapshot mechanism required every contributor to run /magpie-setup
before any framework skill was invocable, and carrying that bootstrap
meant committing a setup skill into three skill directories plus twelve
.gitignore rules. The framework now publishes a plugin marketplace, so
each contributor's agent harness installs and updates it per-user and a
fresh clone needs no setup step at all.
The overrides directory stays committed: the installed plugin still
reads it at run time, so Airflow's triage config and comment templates
keep applying to everyone.
The maintainer triage guide's links pointed into .claude/skills/ in the
framework repo, a layout it abandoned when the skills were split into
per-family plugin directories, so they had been dead for a while.
Claude-Session: https://claude.ai/code/session_01HALJXR27XUFAKof5BFs5rR
* add instructions to not include issue number or anything in PR title
* update hash , example and paragraph
* Update hash
---------
Co-authored-by: Wei Lee <dev@wei-lee.me>
Contributor instructions currently point to a nonexistent Breeze command, preventing developers and coding agents from reproducing selective CI decisions locally.
Signed-off-by: viiccwen <vicwen@apache.org>
The apache-steward framework was renamed to apache-magpie (the GitHub repo
apache/airflow-steward now redirects to apache/magpie). Update the adopter-side
references — AGENTS.md, README.md, the overrides README, the maintainer
PR-triage doc, the .gitignore tracking-issue links, and the committed lock URL
— to the new name so the docs and the pinned fetch URL are canonical.
While updating AGENTS.md, add a "Reviewing pull requests" subsection so agents
default to the locally-installed magpie-pr-management-code-review skill (inline,
file:line-anchored, accept/skip-per-finding) rather than an ad-hoc pass or a
generic review command — following through on guidance the magpie adopt flow
now emits (framework #675, picked up here by refreshing the committed
magpie-setup bootstrap to the current framework tip).
* Reject Conventional Commits style commit messages
Airflow does not use Conventional Commits, but feat:/fix:/chore: style
subjects have crept into commits more and more often in recent months. Add
a commit-msg prek hook that rejects them locally, and a CI step that checks
every commit of a pull request, so contributors get plain-prose subjects
focused on user impact instead.
* Apply suggestions from code review
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
* Check PR title for Conventional Commits and fix applied-suggestion typos
Under "squash and merge" the PR title becomes the commit subject on the
target branch, so CI now validates the PR title (read from an env var to
avoid shell injection) instead of the individual commit subjects, which are
often "fixup"/"apply review" and do not survive the squash. Also fix a
missing comma that concatenated the "bugfix"/"build" types and a stray
non-breaking space that broke test collection.
* Fix yamllint line-length in the PR-title check step
---------
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
Agent-authored GitHub comments that @-mention contributors, committers, or
maintainers generate notification noise and pull people into threads they
have not chosen to join. Default agents to referring to roles, teams, code
ownership, labels, or components instead, reserving individual mentions for
cases a human reviewer has explicitly authorized.
Make explicit on both the authoring and review sides that direct
AirflowException raises are being reduced, not added. New direct usages
are disallowed (already enforced by the check-no-new-airflow-exceptions
prek hook), and the reviewer guidance clarifies that the only acceptable
movement of an AirflowException line is relocating an already-existing
one verbatim during a refactor, which is not a new usage.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
AI-assisted contributions frequently over-comment: multi-line prose
narrating one-line changes, the same rationale duplicated across call
sites, and explanatory comments on tests whose names already convey
intent. The existing review checklist only caught the narrowest case
(a comment restating the very next line), and the write-time coding
standards said nothing at all, so agents had no rule steering them away
from the pattern in the first place.
Add a coding standard directing contributors to comment sparingly and
broaden the review signal so reviewers can flag the wider family of
over-commenting, not just line-restating narration.
Expand dev/breeze/doc/ci/04_selective_checks.md with a "why and how"
explanation of selective checks: the optimisation goal, the design
assumptions (changed files as a risk proxy, conservative-when-in-doubt,
trust earned on main), and a mental model of events -> run mode ->
file-group matches -> outputs.
Add Mermaid diagrams for the decision pipeline (run-mode selection, the
full_tests_needed short-circuit ladder, mode -> jobs, test-type
selection), a table of individually-simple file->effect rules, worked
examples for pull requests / pushes / release-branch pushes / canary
runs, and a troubleshooting section for "my CI run does too much".
Refresh the existing decision-rules list to match current behaviour
(release-branch and only-text PUSH exceptions, API-contract vs API-source
distinction). Add an AGENTS.md rule to keep selective_checks.py and this
doc in sync.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agents (and contributors) have been adding newsfragments for changes that
are not user-facing — build/release tooling, CI, internal refactors —
which then have to be removed in review. Make the expectation explicit: a
newsfragment is only for changes certainly visible to users; when in
doubt, omit it and let a maintainer request one during review.
* Migrate apache-steward framework adoption to Apache Magpie
The framework was renamed apache-steward -> Apache Magpie upstream
(apache/airflow-steward #436-#440). This migrates the adoption to the new
layout via the framework's pre-Magpie -> Magpie migration shim:
- .apache-steward* -> .apache-magpie* (snapshot, committed lock, overrides dir)
- committed setup-steward skill -> magpie-setup
- framework symlinks re-prefixed to magpie-* (list-steward-skills -> magpie-list-skills)
- .gitignore collapsed to a magpie-* glob with a magpie-setup negation
- .pre-commit-config.yaml hook exclusions updated to the new paths
- post-checkout hook + README/AGENTS references updated
Use /magpie-setup for all framework operations from now on.
* magpie-setup: refresh committed bootstrap from airflow-steward main
/magpie-setup upgrade refreshed the snapshot to airflow-steward main
(09902844de, which includes #441) and re-copied the committed bootstrap:
- adopt.md now carries the !/.../magpie-setup gitignore negation (#441)
- drops the duplicate SPDX header the license hook had injected (the skill
is excluded from that hook, so the framework's own header is canonical)
* Remove pr-triage, maintainer-review, pr-stats skills
These three skills were moved into the apache/airflow-steward
framework as pr-management-triage, pr-management-code-review, and
pr-management-stats respectively. They will be provided by the
framework via the adoption flow that follows this commit.
* Adopt apache/airflow-steward via snapshot mechanism
Wire up the apache-steward framework using the install recipe for
git-branch method (Method 3) pinned at branch `main`. The framework
provides the pr-management-* skills (triage, code-review, stats,
mentor) that replace the three local skills removed in the previous
commit.
Committed:
- `.github/skills/setup-steward/` — the only framework artefact that
lives in the adopter repo (per the framework's adoption contract);
every other framework skill is a gitignored symlink into the
gitignored snapshot at `.apache-steward/`.
- `.claude/skills/setup-steward` — symlink into the above (matches
airflow's existing double-symlinked skills convention).
- `.apache-steward.lock` — project pin (git-branch / main). Future
developers cloning this repo run `/setup-steward` to fetch the
snapshot per this pin.
- `.apache-steward-overrides/{README,user}.md` — scaffold for
adopter-specific workflow modifications and per-developer
preferences.
- AGENTS.md — section describing the framework adoption.
Gitignored (build artefacts / per-machine state):
- `.apache-steward/` — the snapshot directory itself.
- `.apache-steward.local.lock` — per-machine record of what was
fetched (compared to the committed lock to detect drift).
- `.claude/skills/{security-*,pr-management-*,setup-isolated-setup-*,setup-shared-config-sync}`
and the matching `.github/skills/*` paths — symlinks created by
`/setup-steward adopt`.
Adopter conventions followed:
- Existing `.claude/skills/*` blanket-ignore preserved; only
`aip-user-stories` (this repo's local skill) and `setup-steward`
(framework entry-point) are explicitly allow-listed.
* Document apache-steward adoption in README contributor section
Add a contributor-facing "Agent-assisted contribution (apache-steward)"
section to README.md that:
- explains the snapshot mechanism (gitignored `.apache-steward/`,
committed `.apache-steward.lock`, committed `setup-steward` skill);
- tells fresh-clone contributors to run `/setup-steward` (or follow
`.claude/skills/setup-steward/`) to populate the framework before
any `pr-management-*` skill is invocable;
- points at `.apache-steward-overrides/` for adopter-specific
workflow modifications and at `apache/airflow-steward` for
framework changes.
The section is placed outside the PyPI sync markers so it does not
leak into PyPI. `doctoc` auto-updated the TOC.
Cross-reference the new section from the matching block in
`AGENTS.md` so agents land on the contributor-facing summary first.
* Add airflow pr-management project-config files
The pr-management-triage / pr-management-stats / pr-management-code-review
skills resolve project-specific values from `<project-config>/`. Copy the
three pr-management-* files from the framework's projects/_template/
directory into `.apache-steward-overrides/` so the skills have:
- committers team handle (`apache/airflow-committers`)
- area label prefix (`area:`)
- project-specific labels (`ready for maintainer review`,
`closed because of multiple quality violations`,
`suspicious changes detected`)
- grace windows for stale sweeps
- comment templates (with airflow doc URLs and AI-attribution footer
wording for airflow's two-stage triage process)
- CI-check-name pattern → category mapping
The framework's templates were authored from Airflow's own values, so
the copies need no further customisation. Future framework refactors
that change the template surface will require manual reconciliation
in these adopter copies.
* Fix static-check failures from steward adoption
- Insert Apache license headers in .apache-steward-overrides/*.md (5 files)
via the insert-license hook auto-fix.
- Normalize SPDX one-liner header indent in
.github/skills/setup-steward/*.md (6 files) to match the
scripts/ci/license-templates/SHORT_LICENSE.md template.
- Exclude .apache-steward-overrides/ and .github/skills/setup-steward/
from lychee and markdownlint — the framework files internally
reference paths under the gitignored .apache-steward/ snapshot,
which doesn't exist in a fresh CI checkout.
- Exclude .github/skills/setup-steward/ from codespell — the
framework's prose uses hyphenated forms (pre-empt, re-use,
Pre-select) that codespell flags as misspellings but shouldn't
be modified locally (they're snapshot files; fixing should happen
upstream in apache/airflow-steward).
- Update contributing-docs/25_maintainer_pr_triage.md links from
old skill names (pr-triage, pr-stats) to framework URLs
(pr-management-triage, pr-management-stats) since the local skills
were removed by this PR.
* Wire steward always-on families + refresh bootstrap skill
The framework's bootstrap version that landed in the previous adoption
commit didn't include Golden Rule 8 (always-on `setup-*` and
`list-steward-*` families are wired unconditionally). Re-running
`/setup-steward upgrade` followed by `/setup-steward` refreshed the
snapshot to apache/airflow-steward@54a2ce4 and surfaced the gap.
Changes in this commit:
- Refresh committed `.github/skills/setup-steward/` from the snapshot.
Adds two new sub-actions the framework grew since the original
adoption: `unadopt.md` and `worktree-init.md`. Re-applies the
SPDX one-liner header the static-check fix from 8b599671bb requires
(the framework ships without it; the airflow static checks need it).
- Extend `.gitignore` to cover the now-symlinked always-on families:
`setup-override-upstream`, `list-steward-*`. The `setup-isolated-setup-*`
patterns were already in place.
Symlinks and the post-checkout hook the framework now installs are
gitignored by design — they don't appear in this diff but are created
on-disk by `/setup-steward` and reproduced on every fresh clone by
re-running the same command per the contributor README section.
providers/ and airflow-ctl/ are both released from main and their release
managers regenerate the changelog from git log — so per-PR newsfragments
are not consumed. Document this in AGENTS.md (root) and providers/AGENTS.md,
and clarify that newsfragments are only used by towncrier-managed
distributions (airflow-core, chart, dev/mypy). Point provider PRs at
docs/changelog.rst and airflow-ctl PRs at RELEASE_NOTES.rst.
Adds a 'GitHub messages drafted by agents' subsection under 'Commits and PRs' that requires any agent-drafted PR/issue comment, PR review, or line-level review comment to end with a 'Drafted-by:' footer. Two forms — one for messages posted without prior human review, one for messages reviewed and approved by a named human maintainer before posting. The footer is in addition to existing PR-body 'Generated-by:' disclosure, not a replacement.
* Run breeze via uvx from the current worktree
Replace `uv tool install -e ./dev/breeze` with a small shim script at
`~/.local/bin/breeze` that runs breeze via `uvx --from <repo>/dev/breeze`.
Each git worktree (including ephemeral worktrees used by coding agents)
gets its own breeze tied to its own sources, removing the
single-global-install ping-pong between checkouts.
Because the shim is a real file on PATH (not a shell function), every
subprocess that invokes `breeze` — pre-commit hooks, CI scripts, dev
tools — sees it just like a `uv tool`-installed binary.
ADR 0017 documents the rationale and supersedes 0016. setup_breeze
writes the shim and refuses to proceed if a legacy `uv tool` / `pipx`
install of breeze is still present (both target the same path). Docs
and in-breeze hints cover both the recommended uvx setup and the
legacy `uv tool` / `pipx` paths. Legacy global installs still work
for contributors who prefer them.
* Recommend setup_breeze shim across contributor docs and AGENTS.md
* Standardize git remote naming on upstream/origin across docs and tooling
Airflow's contributing docs, release runbooks, and agent instructions used
a mix of `apache`, `origin`, and placeholder names for the git remote that
tracks `apache/airflow`. Standardize on `upstream` → `apache/airflow` and
`origin` → the contributor's fork throughout — the two names already
assumed by `dev/sync_fork.sh` and by most beginner-facing docs — so that
contributors, release managers, and agents follow a single convention.
- Add a new "Git remote naming conventions" section to
`contributing-docs/10_working_with_git.rst` with migration commands for
the common legacy layouts (`apache` remote, origin-as-upstream, missing
upstream/fork).
- Update `contributing-docs/10_working_with_git.rst`,
`contributing-docs/18_contribution_workflow.rst`, and all
`dev/README_RELEASE_*.md` runbooks to use `upstream` in examples and
link to the conventions section.
- Update `AGENTS.md` (= `CLAUDE.md`) to document the convention and
instruct agents to surface remote-name mismatches to the user and
propose concrete `git remote rename` commands instead of silently
using whatever names they find.
- Switch breeze CLI defaults and error messages to match: `--remote-name`
now defaults to `upstream` in `release-management start-rc-process`,
`release-management update-constraints`, and
`sbom update-sbom-information`; `breeze ci find-backport-prs` warns
about a missing `upstream` (not `apache`) remote; the fallback list in
`tag_providers` now tries `upstream` first and keeps `apache`/`origin`
for back-compat.
- Regenerate the affected breeze help images and update the
`basic-tests.yml` CI workflow to add `upstream` rather than `apache`.
* Soften "don't push to main" wording in remote conventions section
@o-nikolas pointed out that the new "never push directly to main on
either remote" phrasing in the Git remote naming conventions section
was stricter than the long-standing guidance in
18_contribution_workflow.rst, which explicitly tolerates developing
directly on your fork's main and opening a PR from there.
Align the conventions section with the more permissive stance and
cross-link to the Contribution Workflow step so both places say the
same thing. AGENTS.md keeps the stricter "no pushes to main" rule
because that advice is aimed at automated agents where branch-only
workflows are the right default.
* Update contributing-docs/18_contribution_workflow.rst
Co-authored-by: Dev-iL <6509619+Dev-iL@users.noreply.github.com>
* Address review comments: RST indentation, wording, back-compat choice
- 10_working_with_git.rst: indent the wrapped paragraph and bullet list
under step 1 so Sphinx renders them inside the "1." item instead of
floating outside (flagged by Copilot — the existing step 2+ wraps
already follow this indent rule).
- MANUALLY_GENERATING_IMAGE_CACHE_AND_CONSTRAINTS.md: replace the
double "usually" with a single "Typically" (Dev-iL).
- sbom_commands.py: add ``apache`` back to NotVerifiedBetterChoice's
autocomplete/help list so legacy clones still see it in --help during
the upstream/origin transition. Any remote name is still accepted at
runtime since the type is "NotVerified" (Copilot). Regenerated the
sbom help-text SVGs.
---------
Co-authored-by: Dev-iL <6509619+Dev-iL@users.noreply.github.com>
PRs that cap a dependency, disable a feature behind a flag, or
otherwise defer the real fix were sometimes landing with body text
like "will open a tracking issue" and nothing concrete linking to the
follow-up work — making it easy for the deferred work to be forgotten.
Require the tracking issue to be opened first and linked in the PR
body by number, so the workaround and its removal path are visible to
every reviewer.
* Prefer project-pinned uv and enforce its minimum version in prek hooks
Prek hooks that invoke uv (directly or via breeze) now prefer
`.venv/bin/uv` from the project's main virtualenv over PATH uv so the uv
version is pinned by the project, and verify it satisfies the minimum
declared in `[tool.uv] required-version` in the root pyproject.toml. If
the main-venv uv is missing, the mypy local runner falls back to PATH uv
with a warning. An outdated uv fails fast with upgrade instructions
instead of emitting confusing downstream errors.
* Extend breeze environment checks with uv version verification
`breeze`'s `perform_environment_checks` already validates Docker and
docker-compose versions before running any command; extend it with a
matching `check_uv_version` that reads `[tool.uv] required-version` from
the root `pyproject.toml` and confirms the `uv` on PATH satisfies it,
matching the pattern the prek hooks now use. Users get the same clear
"upgrade uv to X.Y.Z" error from `breeze` commands that they'd see from
a standalone prek hook, instead of a confusing failure deeper in the
shell bootstrap.
* Document that [tool.uv] required-version is not auto-upgraded
`[tool.uv] required-version` is the minimum uv contributors must have
installed, not the exact uv version CI pins to. Bumping it on every uv
release would force every contributor to upgrade uv in lockstep, which
is far more churn than the floor check is worth.
Leave an explicit comment both at the declaration site and in
`scripts/ci/prek/upgrade_important_versions.py` (where `UV_PATTERNS`
could be tempted to pick it up) so a future contributor doesn't wire it
into the automated bump by accident. The breeze/prek uv version checks
already read it dynamically and tolerate a stale floor.
* Move uv version check out of docker utils into environment_check
The uv version check is not docker-specific — it applies to any breeze
entry point that invokes uv, including flows that never talk to Docker.
Move `check_uv_version` and `_read_required_uv_version` from
`docker_command_utils` into the generic `environment_check` module, use
`packaging.version.parse` directly instead of docker-scoped
`compare_version`, and move the unit tests alongside.
`perform_environment_checks` continues to call it after the Docker
checks so the behavior is unchanged.
* Propagate manual [tool.uv] required-version bumps into test fixtures
`[tool.uv] required-version` is bumped manually (by design — see the
comment next to it in pyproject.toml). When it moves, any hard-coded uv
version in tests that represents that floor needs to move too or the
fixtures silently drift.
Add a `sync-uv-min-version-markers` prek hook that reads the current
`required-version` and rewrites any line tagged with a
`# sync-uv-min-version` marker to match. The hook fails (prek
convention) when it makes a change, so the contributor just re-stages.
The existing `check_uv_version` tests carry the marker on their mocked
floor constant so a future bump to `required-version` will propagate
through the marker and keep the success-path assertions in sync.
* Isolate mypy prek hooks, cover all non-provider dirs, and clean up type errors
Each non-provider mypy prek hook now builds and caches its own virtualenv
at .build/mypy-venvs/<hook>/ and its own mypy cache at
.build/mypy-caches/<hook>/. UV_PROJECT_ENVIRONMENT redirects uv away from
the project's .venv, so running the hook never mutates a contributor's
regular development environment while still matching CI's frozen
dependency set. Mypy runs with --follow-imports=silent so each hook only
reports errors for files it owns; transitive code is covered by its own
hook and different venvs no longer produce divergent results on shared
code.
Adds mypy hooks for the non-provider directories that were previously
uncovered: airflow-ctl-tests, helm-tests, airflow-e2e-tests,
task-sdk-integration-tests, docker-tests, kubernetes-tests, and shared.
The mypy-shared hook iterates every shared/<dist> workspace distribution
and builds a separate venv + cache per distribution so each shared
library is type-checked against its own dependency set.
breeze down --cleanup-mypy-cache additionally removes
.build/mypy-venvs/ and .build/mypy-caches/ so all per-hook state is
wiped alongside the existing .mypy_cache and mypy-cache-volume.
Also fixes pre-existing type errors surfaced by the newly added and
cleaned-up checks: platform-specific ignores for Linux-only
os.posix_fadvise in the shared logging helper, narrower types and
type: ignore where appropriate in shared configuration/observability/
timezones/secrets_backend/secrets_masker, Liskov override markers on
the AirflowConfigParser subclass methods, and small correctness fixes
in dev/breeze and the docker-tests / kubernetes-tests helpers so the
full non-provider mypy suite runs clean on macOS and in CI.
* Move mypy prek hooks to their respective distribution configs
The new mypy hooks for airflow-ctl-tests, helm-tests, airflow-e2e-tests,
task-sdk-integration-tests, docker-tests, and kubernetes-tests now live
in each distribution's own .pre-commit-config.yaml, matching the pattern
already used by airflow-core, task-sdk, and airflow-ctl. New .pre-commit-
config.yaml files are added to distributions that didn't have one. prek
auto-discovers nested configs, so the hooks remain part of the default
check set.
mypy-dev (covers dev + scripts), mypy-devel-common, and mypy-shared stay
at the repo root: dev/scripts/devel-common don't have their own configs,
and mypy-shared iterates every shared/<dist> distribution so has no
single home.
* Split mypy-dev and mypy-scripts, each with its own pyproject.toml config
Previously the mypy-dev prek hook ran mypy against dev/ and scripts/ in
a single invocation under the dev project's virtualenv. The two now get
independent hooks — mypy-dev in dev/.pre-commit-config.yaml and
mypy-scripts in scripts/.pre-commit-config.yaml — so each can evolve its
own dependency set and check its own folder.
Copy the full [tool.mypy] section from the root pyproject.toml into both
dev/pyproject.toml and scripts/pyproject.toml so each sub-project owns its
mypy configuration. Paths inside mypy_path are rewritten from
$MYPY_CONFIG_FILE_DIR/ to $MYPY_CONFIG_FILE_DIR/../ so they still resolve
to the repo-root siblings from the sub-project location. The decorator/
outputs plugins are scoped to dev only (scripts does not author DAG code).
mypy_local_folder.py now passes --config-file <project>/pyproject.toml
when the folder maps to one of these sub-project configs, so mypy uses
the sub-project's configuration rather than the root one.
* Teach selective-checks about the new non-provider mypy hooks
Add FileGroupForCi entries and regex patterns for helm-tests,
airflow-e2e-tests, docker-tests, kubernetes-tests, scripts, and shared
Python files, then wire them into skip_prek_hooks so the corresponding
mypy-* prek hook is only kept when its folder changed:
- mypy-scripts (split off from the old combined mypy-dev)
- mypy-airflow-ctl-tests, mypy-helm-tests, mypy-airflow-e2e-tests,
mypy-task-sdk-integration-tests, mypy-docker-tests, mypy-kubernetes-tests
- mypy-shared
Update test_selective_checks.py skip-list constants and per-case inline
skip lists to include the new hooks. Targeted test cases for files under
the new-hook directories override skip-prek-hooks to leave the matching
hook out of the skip set, confirming it will run when its folder changes.
* Trim dev/scripts pyproject mypy_path to just relevant distributions
Drop the 200+ provider path entries that were blindly copied from the root
pyproject.toml. dev and scripts only import from other non-provider
workspace members, so listing every provider src/tests directory under
mypy_path just adds noise. The remaining non-provider entries cover
everything dev or scripts plausibly import from.
* Install mypy into per-hook venvs from uv.lock via a `mypy` dep group
Each non-provider distribution with a mypy prek hook now declares a
`mypy` dependency group in its pyproject.toml resolving to
`apache-airflow-devel-common[mypy]`. mypy_local_folder.py syncs each
dedicated virtualenv with `uv sync --frozen --project <X> --group mypy`
and runs mypy with `uv run --frozen --project <X> --group mypy` — so
mypy and its type stubs come from the workspace uv.lock, not from an
ephemeral `--with` overlay whose resolution is independent of the main
lockfile. uv.lock is refreshed to include the new group.
Covers airflow-core, task-sdk, airflow-ctl, devel-common, dev, scripts,
airflow-ctl-tests, helm-tests, airflow-e2e-tests, task-sdk-integration-
tests, docker-tests, kubernetes-tests, and every shared/<dist>
workspace member.
* Drop mypy_path from dev/scripts pyprojects — venv site-packages is enough
After the switch to installing mypy (and every transitive workspace
dependency) directly into each hook's virtualenv via the `mypy` dep
group, workspace packages like airflow, airflow.sdk, airflowctl,
airflow_breeze, tests_common are all available via the venv's
site-packages. mypy resolves them without needing mypy_path entries,
so drop the copied list and leave a short comment explaining why.
* Split mypy-shared into per-distribution hooks and enforce the pattern
Each shared/<dist> workspace member now owns a mypy-shared-<dist> prek hook
backed by its own shared/<dist>/.pre-commit-config.yaml. The single
mypy-shared iterator is gone — mypy_local_folder.py accepts shared/<dist>
as a first-class folder and the per-hook virtualenv now lives at
.build/mypy-venvs/shared-<dist>/ (slash in the folder name is replaced
with a dash in the venv/cache path).
Adds a new check-shared-mypy-hooks prek hook that fails when a
shared/<dist> workspace member is missing its dedicated .pre-commit-
config.yaml, printing the exact YAML to add. Selective-checks emits one
skip entry per dist, enumerated from shared/ at run time. Contributing
docs cover the two-step process for adding a new shared library.
* Pin minimum_prek_version to 0.3.4 consistently across all configs
All .pre-commit-config.yaml files now require prek >= 0.3.4 (the version
already declared by the root config). Previously the nested configs
pinned a mix of 0.2.0, 0.3.2, and 0.3.4, so a contributor could pass the
root's version check and still trip on stale subproject pins as they
moved between directories.
* Refresh uv.lock after rebase to reflect the `mypy` dep groups
The rebase onto main resolved the uv.lock conflict by taking main's
version, so `uv sync --group mypy` would fail against uv.lock until
the groups added to the per-distribution pyprojects were re-resolved.
Regenerates the lockfile to include them.
* Add explicit selective-checks test for per-shared-dist mypy hook skipping
Verifies that when a file under shared/logging/ changes, only
mypy-shared-logging is kept among the thirteen mypy-shared-* hooks;
all other shared distributions' hooks land in the skip list. Pins the
contract that the runtime enumeration over shared/*/pyproject.toml
works as intended.
* Refresh mypy docs to match the per-hook venv + --group mypy workflow
Fills in the docs that still referenced the pre-split workflow:
- AGENTS.md: mentions `mypy-shared-<dist>` per shared workspace member and
the `uv sync --group mypy` install path for mypy itself.
- scripts/ci/prek/AGENTS.md: clarifies that non-provider mypy hooks run
locally through mypy_local_folder.py (Breeze image only needed for the
providers hook).
- dev/breeze/doc/03_developer_tasks.rst: renames stale `mypy-airflow` to
`mypy-airflow-core`, and expands the cache note to cover the per-hook
virtualenvs and caches under .build/.
- dev/breeze/doc/ci/04_selective_checks.md: expands the file-group and
skip-reason lists so every new mypy hook (scripts, task-sdk, airflow-ctl,
the six test-dir hooks, and mypy-shared-<dist> enumerated at runtime)
is documented.
* Rename mypy_local_folder.py to run_mypy_full_dist_local_venv_or_breeze_in_ci.py
Updates every .pre-commit-config.yaml entry and prose references so they
point at the new script name. Two shared configs use YAML folded-scalar
entries to stay under the 110-char yamllint limit; updates the validation
script's expected template to match.
Mypy checks for non-provider projects now synchronize the local
virtualenv with uv.lock (uv sync --frozen) before running, so contributors
see the same dependency set CI uses and avoid results that drift from CI.
The update-uv-lock prek hook now runs with --frozen, so pyproject.toml
changes that would touch uv.lock fail the hook and require an explicit
uv lock + commit instead of silently rewriting the lock during a commit.
* Docs: Add JWT authentication docs and strengthen security model
Add comprehensive JWT token authentication documentation covering both
the REST API and Execution API flows, including token structure, timings,
refresh mechanisms, and the DFP/Triggerer in-process bypass.
Update the security model to:
- Document current isolation limitations (DFP/Triggerer DB access,
shared Execution API resources, multi-team not guaranteeing task-level
isolation)
- Add deployment hardening guidance (per-component config, asymmetric
JWT keys, env vars with PR_SET_DUMPABLE protection)
- Add "What is NOT a security vulnerability" section covering all
categories from the security team's response policies
- Fix contradicting statements across docs that overstated isolation
guarantees or recommended sharing all config across components
Update AGENTS.md with security model awareness so AI agents performing
security research distinguish intentional design choices from actual
vulnerabilities.
* Fix spelling errors and use 'potentially' for DFP/Triggerer access
- Add dumpable, sandboxing, unsanitized, XSS to spelling wordlist
- Use 'potentially' consistently when describing Dag File Processor
and Triggerer database access and JWT authentication bypass, since
these are capabilities that Dag author code could exploit rather
than guaranteed behaviors of normal operation
* Add prek hook to validate security doc constants against config.yml
New hook `check-security-doc-constants` validates that:
- [section] option references in security RST files match config.yml
- AIRFLOW__X__Y env var references correspond to real config options
- Default values in doc tables match config.yml defaults
- Sensitive config variables are listed (warning, not error, since
the list is documented as non-exhaustive)
Loads both airflow-core config.yml and provider.yaml files to cover
all config sections (including celery, sentry, workers, etc.).
Runs automatically when config.yml or security RST docs are modified.
* Expand sensitive vars to full list with component mapping and auto-update
Update security_model.rst sensitive config variables section:
- List ALL sensitive vars from config.yml and provider.yaml files
- Core vars organized in a table with "Needed by" column mapping each
var to the components that require it (API Server, Scheduler, Workers,
Dag File Processor, Triggerer)
- Provider vars in a separate table noting they should only be set where
the provider functionality is needed
- Tables are auto-generated between AUTOGENERATED markers
Update prek hook to auto-update the sensitive var tables:
- Reads config.yml and all provider.yaml files
- Generates RST list-table content for core and provider sensitive vars
- Replaces content between markers on each run
- Warns when new sensitive vars need component mapping added to the hook
- Validates [section] option and AIRFLOW__X__Y references against config
- Skips autogenerated sections when checking env var references
* Clarify software guards vs intentional access in DFP/Triggerer
Address issues raised in security discussion about the gap between
Airflow's isolation promises and reality:
- Clearly distinguish software guards (prevent accidental DB access)
from the inability to prevent intentional malicious access by code
running as the same Unix user as the parent process
- Document the specific mechanisms: /proc/PID/environ, config files,
_CMD commands, secrets manager credential reuse
- Clarify that worker isolation is genuine (no DB credentials at all)
while DFP/Triggerer isolation is software-level only
- Add Unix user impersonation as a deployment hardening measure
- Document strategic (API-based DFP/Triggerer) and tactical (user
impersonation) planned improvements
- Add warning about sensitive config leakage through task logs
- Add guidance to restrict task log access
* Docs: Improve security docs wording, extract workload isolation, recommend DagBundle
- Reword DFP/Triggerer descriptions to clarify software guards vs intentional bypass
- Extract workload isolation section from jwt_token_authentication into workload.rst
- Recommend Dag Bundle mechanism (GitDagBundle) for DAG synchronization
- Fix typo in public-airflow-interface.rst and broken backtick in jwt_token_authentication.rst
- Update cross-references between security docs
Non-provider mypy checks (airflow-core, task-sdk, airflow-ctl, dev, scripts,
devel-common) now run locally via uv as regular prek hooks in the pre-commit
stage, instead of running as separate mypy CI jobs in the CI image checks
workflow. This means they run as part of the regular static checks job in CI
and automatically on every local commit.
The folder-level mypy checks (which check entire directories at once for
comprehensive cross-file type checking) replace the previous file-level
incremental checks.
Provider mypy checks still run via breeze as a dedicated CI job, now embedded
directly in the main CI workflow (ci-amd-arm.yml) instead of being dispatched
through the ci-image-checks reusable workflow.
The selective checks logic skips non-provider mypy hooks when their relevant
files haven't changed, unless devel-common/pyproject.toml changes on main
(which affects all mypy configurations).
Shahar Epstein
Andrew Chang
Ash Berlin-Taylor
LI,JHE-CHEN
Pratiksha BadhekaandWei Lee
Vic Wen
Amogh Desai
Tzu-ping Chung
Pierre Jeambrun
Vincent
Jed Cunningham
Jason(Zhe-You) Liu
Wei Lee
hojeong park
Jens Scheffler