mirror of
https://github.com/apache/airflow.git
synced 2026-11-04 07:12:48 +00:00
Add Git-Sync to GitDagBundle migration guide for the Helm chart (#73082)
* Add migration guide from Git-Sync sidecar to GitDagBundle The Git-Sync sidecar (dags.gitSync) is planned for removal in Helm Chart 2.0 in favour of the Airflow-native GitDagBundle. This adds a chart docs page mapping the old dags.gitSync values to their dagProcessor.dagBundleConfigList equivalents, with authentication patterns and a step-by-step migration. * Address reviewer feedback: remove inaccurate Git-Sync claim Veyu noted Git-Sync could checkout any commit via rev/ref params, so the statement that it always checked out the latest commit was not accurate. Removed that sentence per the preferred fix, keeping the GitDagBundle versioning note. * docs: address review feedback on Git-Sync to GitDagBundle migration guide - Fix spellcheck: favour -> favor (en_US) - Generalize Git-Sync sidecar description (persistence-setting dependent) - Prereq: latest Helm Chart 1.x release with Airflow 3 - credentialsSecret row lists both GITSYNC_* and GIT_SYNC_* key names - Clarify what kwargs.git_conn_id refers to - Note the Airflow Connection UI/CLI alternative to extraSecrets/extraEnv --------- Co-authored-by: Przemysław Mirowski <17602603+Miretpl@users.noreply.github.com>
This commit is contained in:
Rakesh PaiandPrzemysław Mirowski authored and 
GitHub committed
1 parent
3edd0521b6
commit
d3e29e1c6d
2 files changed
+227
No files matched your search
@@ -46,6 +46,7 @@ Helm Chart for Apache Airflow
|
||||
production-guide
|
||||
service-account-token-examples
|
||||
upgrading-to-airflow-3
|
||||
Migrating from Git-Sync to GitDagBundle<migrate-to-git-dag-bundle>
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
.. Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
.. http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
.. Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
|
||||
.. _git_sync_to_git_dag_bundle:
|
||||
|
||||
Migrating from Git-Sync to GitDagBundle
|
||||
=======================================
|
||||
|
||||
This guide is for users of the Git-Sync sidecar (``dags.gitSync``) in Helm Chart 1.x.
|
||||
The Git-Sync sidecar is planned for removal in Helm Chart 2.0 in favor of the
|
||||
Airflow-native :class:`~airflow.providers.git.bundles.git.GitDagBundle`.
|
||||
|
||||
What is changing
|
||||
----------------
|
||||
|
||||
Airflow components pods keep a shared DAG folder up to date from a git repository.
|
||||
|
||||
With ``GitDagBundle``, syncing moves into Airflow itself. The Dag Processor clones
|
||||
the repository once into a local bare repository and refreshes it every
|
||||
``refresh_interval`` seconds. Task execution resolves DAG code from versioned
|
||||
bundles, so workers always run the code version the Dag Processor recorded. No
|
||||
sidecar containers and no shared ``ReadWriteMany`` volume are needed for syncing.
|
||||
|
||||
Differences to be aware of before migrating:
|
||||
|
||||
* Git-Sync synced a single repository per deployment (multiple repositories required
|
||||
an "umbrella" repository with submodules). ``dagProcessor.dagBundleConfigList``
|
||||
accepts multiple bundles, so each repository simply becomes its own bundle.
|
||||
* ``GitDagBundle`` supports versioning: changing a SHA-pinned ``tracking_ref`` is a configuration change and
|
||||
only takes effect once the Dag Processor restarts and reloads its configuration.
|
||||
Promoting to a *new* SHA can also require the bundle's local storage to be
|
||||
cleared first (for example by letting the pod be recreated) — see
|
||||
`GH-71388 <https://github.com/apache/airflow/issues/71388>`_.
|
||||
* ``dags.persistence`` was often enabled only to give Git-Sync a shared volume. It
|
||||
is not needed for ``GitDagBundle`` and can be disabled if nothing else uses it.
|
||||
|
||||
Prerequisites
|
||||
-------------
|
||||
|
||||
* Latest Helm Chart 1.x release with Airflow 3 (DAG bundles are an Airflow 3 feature).
|
||||
* The ``apache-airflow-providers-git`` provider package must be installed in the
|
||||
image you deploy, along with the ``git`` binary (the ``git`` binary is
|
||||
pre-installed in the official ``apache/airflow`` images starting from Airflow
|
||||
3.0.2). If you build a custom slim image, add the provider to it.
|
||||
|
||||
Value mapping
|
||||
-------------
|
||||
|
||||
.. list-table::
|
||||
:header-rows: 1
|
||||
:widths: 40 60
|
||||
|
||||
* - Git-Sync (``dags.gitSync.*``)
|
||||
- GitDagBundle (``dagProcessor.dagBundleConfigList[].kwargs``)
|
||||
* - ``repo`` — the clone URL
|
||||
- ``repo_url``. Alternatively, store the repository URL in the ``host`` field
|
||||
of a ``git`` connection and reference it with ``git_conn_id``.
|
||||
* - ``branch`` (git-sync v3) / ``ref`` (git-sync v4) — branch, tag or hash to
|
||||
check out
|
||||
- ``tracking_ref`` — accepts a branch, tag or full commit SHA.
|
||||
* - ``rev`` (git-sync v3)
|
||||
- ``tracking_ref``. To follow the tip of a branch, set ``tracking_ref`` to
|
||||
the branch name.
|
||||
* - ``subPath`` — subdirectory with DAGs
|
||||
- ``subdir``
|
||||
* - ``period`` — sync interval as a Go duration string (for example ``"5s"``)
|
||||
- ``refresh_interval`` — sync interval in seconds (integer, for example
|
||||
``300``)
|
||||
* - ``credentialsSecret`` (``GITSYNC_USERNAME``, ``GIT_SYNC_USERNAME``, ``GITSYNC_PASSWORD``, ``GIT_SYNC_PASSWORD``)
|
||||
- ``git_conn_id`` — an Airflow ``git`` connection whose login/password hold
|
||||
the username and access token. See :ref:`git_sync_to_git_dag_bundle:auth`
|
||||
below.
|
||||
* - ``sshKeySecret`` / ``sshKey`` (``gitSshKey``)
|
||||
- ``git_conn_id`` — a ``git`` connection with the SSH private key in the
|
||||
``private_key`` extra (inline key) or ``key_file`` extra (path to a key
|
||||
file).
|
||||
* - ``knownHosts``
|
||||
- ``git_conn_id`` — a ``git`` connection with the ``known_hosts_file`` and
|
||||
``strict_host_key_checking`` extras.
|
||||
* - ``depth``, ``maxFailures``, probes, ``resources``, ``env``
|
||||
- No equivalent. There are no sidecar containers to tune: ``GitDagBundle``
|
||||
keeps one full bare clone and fetches from it, and refresh failures are
|
||||
retried internally.
|
||||
|
||||
.. _git_sync_to_git_dag_bundle:auth:
|
||||
|
||||
Authentication
|
||||
--------------
|
||||
|
||||
``GitDagBundle`` authenticates through a standard Airflow connection of type
|
||||
``git``, selected via the bundle's ``git_conn_id`` kwarg (i.e. the
|
||||
``dagProcessor.dagBundleConfigList[].kwargs.git_conn_id`` value), or needs no
|
||||
connection at all for public repositories when ``repo_url`` is given directly.
|
||||
|
||||
For a private repository over HTTPS with a personal access token, create the
|
||||
connection from a Kubernetes secret managed by the chart:
|
||||
|
||||
.. code-block:: yaml
|
||||
:caption: override-values.yaml
|
||||
|
||||
extraSecrets:
|
||||
'{{ .Release.Name }}-git-connections':
|
||||
stringData:
|
||||
# The connection id is "git_default" (env var name is upper-cased).
|
||||
# login/password become the token credentials; repo_url in the bundle
|
||||
# kwargs selects the repository.
|
||||
AIRFLOW_CONN_GIT_DEFAULT: 'git://oauth2:<your-token>@github.com/your-org/dags-repo.git'
|
||||
|
||||
Then mount it into the pods that need it (the Dag Processor at minimum):
|
||||
|
||||
.. code-block:: yaml
|
||||
:caption: override-values.yaml
|
||||
|
||||
dagProcessor:
|
||||
extraEnv:
|
||||
- name: AIRFLOW_CONN_GIT_DEFAULT
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: '{{ .Release.Name }}-git-connections'
|
||||
key: AIRFLOW_CONN_GIT_DEFAULT
|
||||
|
||||
.. note::
|
||||
|
||||
The connection ``host`` field is ignored when ``repo_url`` is set in the
|
||||
bundle kwargs — only the credentials from the connection are used. For the
|
||||
full list of connection extras (SSH keys, known hosts, proxy, GitHub App
|
||||
auth), see :doc:`apache-airflow-providers-git:bundles/index`.
|
||||
|
||||
Instead of injecting the connection through ``extraSecrets``/``extraEnv``,
|
||||
you can create the same ``git`` connection directly in the Airflow UI
|
||||
(**Admin** → **Connections**) or with ``airflow connections add`` — the Dag
|
||||
Processor resolves any connection with a matching conn id from the metadata
|
||||
database.
|
||||
|
||||
Step-by-step migration
|
||||
----------------------
|
||||
|
||||
1. Translate your ``dags.gitSync`` settings using the mapping table above and add
|
||||
a bundle to ``dagProcessor.dagBundleConfigList``:
|
||||
|
||||
.. code-block:: yaml
|
||||
:caption: override-values.yaml
|
||||
|
||||
dagProcessor:
|
||||
dagBundleConfigList:
|
||||
- name: dags-repo
|
||||
classpath: "airflow.providers.git.bundles.git.GitDagBundle"
|
||||
kwargs:
|
||||
repo_url: "https://github.com/your-org/dags-repo.git"
|
||||
tracking_ref: "main" # was dags.gitSync.branch / ref
|
||||
subdir: "dags" # was dags.gitSync.subPath
|
||||
refresh_interval: 300 # was dags.gitSync.period: "5s"
|
||||
git_conn_id: "git_default" # only needed for private repos
|
||||
|
||||
The chart renders this list into
|
||||
``config.dag_processor.dag_bundle_config_list`` automatically.
|
||||
|
||||
2. If nothing else uses the shared DAG volume, disable the pieces Git-Sync
|
||||
needed:
|
||||
|
||||
.. code-block:: yaml
|
||||
:caption: override-values.yaml
|
||||
|
||||
dags:
|
||||
persistence:
|
||||
enabled: false
|
||||
gitSync:
|
||||
enabled: false
|
||||
|
||||
Keep the default ``dags-folder`` ``LocalDagBundle`` in the bundle list only if
|
||||
you still bake DAGs into the image; otherwise remove it.
|
||||
|
||||
3. Upgrade the release:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
helm upgrade --install airflow apache-airflow/airflow \
|
||||
--namespace airflow -f override-values.yaml
|
||||
|
||||
4. Verify the bundle is picked up. The Dag Processor log should show the bundle
|
||||
being configured and cloned; in the UI, DAGs appear with the bundle name
|
||||
(``dags-repo``) as their source. Trigger a test DAG run to confirm tasks
|
||||
resolve code from the bundle.
|
||||
|
||||
Multiple repositories
|
||||
---------------------
|
||||
|
||||
Unlike Git-Sync, which supported a single repository, you can declare one bundle
|
||||
per repository:
|
||||
|
||||
.. code-block:: yaml
|
||||
:caption: override-values.yaml
|
||||
|
||||
dagProcessor:
|
||||
dagBundleConfigList:
|
||||
- name: team-a-dags
|
||||
classpath: "airflow.providers.git.bundles.git.GitDagBundle"
|
||||
kwargs:
|
||||
repo_url: "https://github.com/your-org/team-a-dags.git"
|
||||
tracking_ref: "main"
|
||||
subdir: "dags"
|
||||
refresh_interval: 300
|
||||
- name: team-b-dags
|
||||
classpath: "airflow.providers.git.bundles.git.GitDagBundle"
|
||||
kwargs:
|
||||
repo_url: "https://github.com/your-org/team-b-dags.git"
|
||||
tracking_ref: "main"
|
||||
subdir: "dags"
|
||||
refresh_interval: 300
|
||||
|
||||
For the complete ``GitDagBundle`` reference, see
|
||||
:doc:`apache-airflow-providers-git:bundles/index`.
|
||||
Reference in new issue
Block a user