mirror of
https://github.com/apache/airflow.git
synced 2026-11-04 07:12:48 +00:00
* Document fork CI for contributors and maintainers Contributors need to know how to set up CI in their fork, how to check that setup, and how to join or leave the pilot, and maintainers need to know which parts of a gated pull request's lifecycle the reconciler owns so triage does not fight it. The previous one-line advice to enable GitHub Actions in the fork did not say any of that. Generated-by: Claude Opus 5 * Say when fork CI applies in the contributor docs While the gate is off, a contributor outside the pilot who enabled fork CI as the docs said got CI twice on every pull request. The docs now say fork CI applies to pilot participants and use fork ci pull requests until it is switched on for everyone, and the maintainer docs cover the needs project ci label, the CI passed status and why removing fork ci required is not enough. Generated-by: Claude Opus 5 * Describe apache/airflow push runs after release-branch CI moved to a schedule ci-arm.yml no longer lists the push branches (#75420). Point to CANONICAL_PUSH_BRANCH_FILTERS and say that the incremental push base is only used in apache/airflow. Generated-by: Claude Opus 5 * Fix the fork CI section title level and label wording in the contributor docs The new subsection used a title style the file had not established, which made Newsfragments a level-4 section and broke links to it. Also say when a pull request the author drafted gets ready for maintainer review, use a placeholder pilot instant, and describe the fork compare base as breeze's default branch rather than the branch name. Generated-by: Claude Opus 5
242 lines
8.8 KiB
ReStructuredText
242 lines
8.8 KiB
ReStructuredText
.. Licensed to the Apache Software Foundation (ASF) under one
|
||
or more contributor license agreements. See the NOTICE file
|
||
distributed with this work for additional information
|
||
regarding copyright ownership. The ASF licenses this file
|
||
to you under the Apache License, Version 2.0 (the
|
||
"License"); you may not use this file except in compliance
|
||
with the License. You may obtain a copy of the License at
|
||
|
||
.. http://www.apache.org/licenses/LICENSE-2.0
|
||
|
||
.. Unless required by applicable law or agreed to in writing,
|
||
software distributed under the License is distributed on an
|
||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||
KIND, either express or implied. See the License for the
|
||
specific language governing permissions and limitations
|
||
under the License.
|
||
|
||
Your First Airflow Pull Request — 15-Minute Guide
|
||
=================================================
|
||
|
||
.. contents:: On this page
|
||
:local:
|
||
:depth: 1
|
||
|
||
|
||
Purpose
|
||
-------
|
||
This page walks **new contributors** through opening their first
|
||
Apache Airflow pull request (PR) in about 15 minutes. We present *one*
|
||
local option (Breeze) and *one* fully-hosted option (GitHub Codespaces).
|
||
Everything else lives in the advanced guides.
|
||
|
||
Prerequisites
|
||
-------------
|
||
* GitHub account
|
||
* `Fork <https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo?tool=webui>`_ `apache/airflow <https://github.com/apache/airflow>`__
|
||
* `Basic Git <https://docs.github.com/en/get-started/git-basics/set-up-git>`__ (**only** required for the Breeze path below)
|
||
|
||
For Breeze (local development):
|
||
|
||
* `Docker Desktop <https://www.docker.com/products/docker-desktop/>`__
|
||
* `Podman <https://podman.io/>`__ is a license-friendly alternative to Docker Desktop, but it is
|
||
**not yet fully supported** by Breeze. Breeze will detect Podman and print a warning, and you
|
||
may run into issues that do not occur with Docker. Docker Desktop is recommended unless you are
|
||
comfortable troubleshooting Podman-specific problems yourself. Track full support in
|
||
`#54894 <https://github.com/apache/airflow/issues/54894>`_.
|
||
* `Docker Compose <https://docs.docker.com/compose/install/>`__
|
||
* `uv <https://github.com/astral-sh/uv>`__, which is a fast, reliable package manager that you'll use to install other developer tools to make contributing to Airflow easier.
|
||
|
||
.. code-block:: bash
|
||
|
||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||
* `Prek <https://github.com/j178/prek>`__, which runs Airflow's required code-quality checks (formatting, linting, and bug-spotting) before you commit, helping save contributors and committers time during the pull request process.
|
||
|
||
.. code-block:: bash
|
||
|
||
uv tool install prek
|
||
|
||
* 4GB RAM, 40GB disk space, and at least 2 CPU cores
|
||
|
||
.. note::
|
||
Docker installation varies by OS. See the `full guide <03_contributors_quick_start.rst#local-machine-development>`_
|
||
for Ubuntu, macOS, and Windows instructions. Podman can be installed the same way, but since it is not
|
||
yet fully supported by Breeze, use it at your own risk and prefer Docker if you run into problems.
|
||
|
||
Option A – Breeze on Your Laptop
|
||
--------------------------------
|
||
1. **Clone your fork**
|
||
|
||
.. code-block:: bash
|
||
|
||
git clone https://github.com/<you>/airflow.git
|
||
cd airflow
|
||
git remote add upstream https://github.com/apache/airflow.git
|
||
|
||
Before your first push, run ``breeze ci audit`` (after installing Breeze below) to check that your
|
||
remotes and the CI in your fork are set up; see
|
||
`CI in your fork <18_contribution_workflow.rst#ci-in-your-fork>`__.
|
||
|
||
2. **Set up Git hooks and install Breeze**
|
||
|
||
Run these commands from the root of your cloned Airflow repository:
|
||
|
||
.. code-block:: bash
|
||
|
||
prek install -f
|
||
prek install -f --hook-type pre-push
|
||
./scripts/tools/setup_breeze
|
||
|
||
3. Setup your idea workspace to detect project src/ and tests/ folders as source roots.
|
||
|
||
.. code-block:: bash
|
||
|
||
# For IntelliJ IDEA and PyCharm
|
||
uv run dev/ide_setup/setup_idea.py
|
||
|
||
# For VS Code
|
||
uv run dev/ide_setup/setup_vscode.py
|
||
|
||
4. Create a branch for your change
|
||
|
||
.. code-block:: bash
|
||
|
||
git checkout -b <your-branch-name>
|
||
|
||
5. **Start the development container** (first run builds the image)
|
||
|
||
.. code-block:: bash
|
||
|
||
breeze start-airflow
|
||
|
||
The command starts a shell and launches multiple terminals using ``mprocs`` by default
|
||
and launches all Airflow necessary components in those terminals.
|
||
You can also choose to use ``tmux`` via the ``--terminal-multiplexer tmux`` option.
|
||
If you are using tmux, check out this cheat sheet to learn more about its commands: https://tmuxcheatsheet.com/.
|
||
|
||
Now you can also access the Airflow UI on your local machine at `http://localhost:28080 <http://localhost:28080>`_ with user name ``admin`` and password ``admin``.
|
||
|
||
To exit breeze, press ``q`` in the ``mprocs`` interface (or in any of the tmux panes) and clean up the resources by running the following command:
|
||
|
||
.. code-block:: bash
|
||
|
||
breeze down
|
||
|
||
**Working with Dags in Breeze:**
|
||
|
||
- **Adding your own Dags**: Place your Dag files in the ``/files/dags/`` directory in your local Airflow repository. This directory is automatically mounted into the Breeze container and your Dags will be visible in the Airflow UI.
|
||
|
||
- **Loading example Dags**: Use the ``--load-example-dags`` flag to load all example Dags from the repository:
|
||
|
||
.. code-block:: bash
|
||
|
||
breeze start-airflow --load-example-dags
|
||
|
||
This flag enables configuration to load example Dags when starting Airflow, which is useful for exploring Airflow's capabilities and testing.
|
||
|
||
6. **Set up virtual environment and install dependencies** - To run prek locally
|
||
|
||
Install airflow dependencies from this `guide <03_contributors_quick_start.rst#setting-up-virtual-env>`__.
|
||
|
||
7. **Make a tiny change** – e.g. fix a typo in docs
|
||
|
||
8. **Run local checks**
|
||
|
||
.. code-block:: bash
|
||
|
||
prek --all-files
|
||
|
||
9. **Run tests**
|
||
|
||
Run tests related to your change **before** pushing:
|
||
|
||
.. code-block:: bash
|
||
|
||
# Example: run core tests
|
||
breeze testing core-tests
|
||
|
||
Run ``breeze testing --help`` to see all available test groups.
|
||
For more on testing, see the `Testing Guide <09_testing.rst>`_.
|
||
|
||
10. **Commit & push**
|
||
|
||
.. code-block:: bash
|
||
|
||
git commit -am "fix typo in README"
|
||
git push -u origin <your-branch-name>
|
||
|
||
11. **Open the PR** – GitHub shows a "Compare & pull request" button.
|
||
|
||
*Syncing your branch*
|
||
|
||
.. code-block:: bash
|
||
|
||
git fetch upstream && git rebase upstream/main && git push --force-with-lease
|
||
|
||
Option B – One-Click GitHub Codespaces
|
||
---------------------------------------
|
||
|
||
1. On **your fork**, click *Code → Codespaces → New codespace*.
|
||
2. Wait for the VS Code web IDE to appear. A terminal opens automatically.
|
||
3. Install Docker Buildx and Docker Compose (required for Breeze)
|
||
|
||
.. code-block:: bash
|
||
|
||
mkdir -p ~/.docker/cli-plugins
|
||
|
||
# Install Docker Buildx
|
||
BUILDX_VERSION=v0.16.2
|
||
curl -SL "https://github.com/docker/buildx/releases/download/${BUILDX_VERSION}/buildx-${BUILDX_VERSION}.linux-amd64" -o ~/.docker/cli-plugins/docker-buildx
|
||
chmod +x ~/.docker/cli-plugins/docker-buildx
|
||
docker buildx version
|
||
|
||
# Install Docker Compose v2
|
||
curl -SL "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s | tr '[:upper:]' '[:lower:]')-$(uname -m)" -o ~/.docker/cli-plugins/docker-compose
|
||
chmod +x ~/.docker/cli-plugins/docker-compose
|
||
docker compose version
|
||
|
||
4. Verify Docker is accessible
|
||
|
||
.. code-block:: bash
|
||
|
||
docker info
|
||
|
||
If ``docker info`` fails, try rebuilding the Codespace container
|
||
(Command Palette → *Codespaces: Rebuild Container*) or restarting
|
||
the Codespace from the GitHub Codespaces dashboard.
|
||
|
||
5. Install Breeze and start the development container
|
||
|
||
.. code-block:: bash
|
||
|
||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||
uv tool install prek
|
||
prek install -f
|
||
prek install -f --hook-type pre-push # for running mypy checks when pushing to repo
|
||
./scripts/tools/setup_breeze
|
||
uv run dev/ide_setup/setup_vscode.py
|
||
breeze start-airflow
|
||
|
||
6. Edit a file in the editor, save, and commit via the Source Control sidebar.
|
||
Push when prompted.
|
||
|
||
7. Press **Create pull request** when GitHub offers.
|
||
|
||
|
||
|
||
Review & Merge
|
||
--------------
|
||
Respond to reviewer comments, push updates (same commands as above). Once
|
||
CI is green and reviews are ✅, a committer will merge. 🎉
|
||
|
||
Example Dag Review Checklist
|
||
----------------------------
|
||
|
||
To help maintain consistency and quality across example Dags,
|
||
please refer to the `example Dag review checklist <28_example_dag_review_checklist.rst>`_.
|
||
|
||
Next Steps
|
||
----------
|
||
* Need a full development environment? See the `Development Environments Guide <https://github.com/apache/airflow/blob/main/contributing-docs/06_development_environments.rst>`_.
|
||
* Learn about our contribution workflow? Checkout the `Contribution Workflow Guide <https://github.com/apache/airflow/blob/main/contributing-docs/18_contribution_workflow.rst>`_.
|