mirror of
https://github.com/apache/airflow.git
synced 2026-11-04 07:12:48 +00:00
* Add consolidated Language SDK compatibility matrix Users comparing Language SDKs had to read each SDK's README, and the per-SDK manifests lagged behind what the SDKs actually ship. One generated matrix, split by authoring mode, shows what each SDK supports for a Dag definition and for a Python Dag with stub TaskHandler. * Add callbacks, listeners and provider operators to the SDK matrix Users comparing Language SDKs could not see where callbacks, Airflow's listener API and Python provider operators stand, because the matrix had no dimension for any of them. Callbacks ask different things of an SDK per authoring mode, so they are declared per mode like the other shared capabilities. * Address review on the Language SDK matrix generator The umbrella capability and the authoring mode both read `dag-definition`, which gives one name two meanings in the same manifest. A row's kind also belongs in its entry key: state and capability names happen not to collide today, so the lookup worked by accident rather than by design. * Rename the dynamic-task-mapping dimension to mapped-tasks The dimension is about whether a task can be a mapped task, which the shorter name says directly and keeps the matrix row readable next to the other task-level dimensions. * Split callbacks into Dag-level and task-level dimensions The callbacks of a Python Dag run whichever SDK executes its stub tasks, so ``dag-callbacks`` is a Dag-definition concern only, while ``task-callbacks`` stays declared per authoring mode. Drop ``provider-operators`` until ADR-0009 is accepted, and reuse the shared ``capabilities_doc`` fixture in the per-SDK README tests. * Fix the hook attribution in the matrix generation diagram ``index.rst`` is written by ``update-lang-sdk-central-matrix`` from every SDK in ``LANG_SDKS``, not by the per-SDK README hook. ---------
642 lines
27 KiB
ReStructuredText
642 lines
27 KiB
ReStructuredText
.. Licensed to the Apache Software Foundation (ASF) under one
|
|
or more contributor license agreements. See the NOTICE file
|
|
distributed with this work for additional information
|
|
regarding copyright ownership. The ASF licenses this file
|
|
to you under the Apache License, Version 2.0 (the
|
|
"License"); you may not use this file except in compliance
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
.. http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
.. Unless required by applicable law or agreed to in writing,
|
|
software distributed under the License is distributed on an
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations
|
|
under the License.
|
|
|
|
Creating a new Language SDK
|
|
===========================
|
|
|
|
Starting from 3.3, the standard Airflow workers can run task code implemented in
|
|
languages other than Python, using a foreign Language SDK. This document
|
|
describes how a new Language SDK can be contributed, so Airflow can execute
|
|
tasks implemented in the language. It is also the normative *conformance
|
|
specification* a Language SDK is expected to meet: the `Conformance`_ section
|
|
defines the exact TaskInstance states and capabilities an SDK MUST, SHOULD, or
|
|
MAY support.
|
|
|
|
Two components are needed for Airflow to understand how to execute such a task:
|
|
|
|
* A **coordinator implemented in Python** for Airflow to understand how to wire
|
|
up the foreign package.
|
|
* A **language SDK implemented in the target language** to talk to the
|
|
coordinator.
|
|
|
|
The two components are largely independent: the coordinator decides *how* to
|
|
start the foreign runtime and *how* to communicate with it, and the language SDK
|
|
implements the other end of whatever protocol the coordinator chooses. There is
|
|
no single mandated communication mechanism.
|
|
|
|
For a diagram-driven overview of how a task runs across the Python and Coordinator
|
|
paths, see `Task execution architecture <31_task_execution_architecture.rst>`__.
|
|
|
|
.. contents:: Table of Contents
|
|
:depth: 2
|
|
:local:
|
|
|
|
|
|
Python Coordinator
|
|
------------------
|
|
|
|
The coordinator is a Python class that Airflow calls when a stub task with a
|
|
matching queue is triggered. Its sole required method is ``execute_task``, which
|
|
must start the foreign runtime, hand off the task, and return the final task
|
|
state. The base class is
|
|
:class:`airflow.sdk.execution_time.coordinator.BaseCoordinator`.
|
|
|
|
A coordinator can communicate with the foreign runtime in any way it chooses.
|
|
This can be a subprocess over TCP sockets, a gRPC server, shared memory, a
|
|
message queue, or anything else. The choice of transport is entirely up to the
|
|
coordinator and its SDK counterpart.
|
|
|
|
Choosing an implementation path
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
In practice, almost all coordinators will fall into one of the following
|
|
categories. Choose the one that fits the target runtime:
|
|
|
|
Use :class:`~airflow.sdk.coordinators.executable.ExecutableCoordinator`
|
|
If the language SDK compiles to a self-contained native executable, the
|
|
existing ``ExecutableCoordinator`` can discover and launch it with zero
|
|
Python code. The executable just needs to carry an ``AFBNDL01`` metadata
|
|
trailer (see `Native Executable Bundle Format`_ below). This is the simplest
|
|
path since you would not need to write any Python code.
|
|
|
|
Subclass :class:`~airflow.sdk.coordinators._subprocess.SubprocessCoordinator`
|
|
If a command is needed (e.g. `java` for JRE, `node` for Node), but
|
|
the produced runtime can communicate over TCP, consider subclassing
|
|
``SubprocessCoordinator`` and implement ``_build_execute_task_command``.
|
|
The base class handles all TCP socket lifecycle, process ownership
|
|
verification, startup draining, and teardown.
|
|
|
|
Subclass ``BaseCoordinator`` directly
|
|
For runtimes that use a completely different transport (gRPC, shared memory,
|
|
a persistent daemon, etc.), subclass ``BaseCoordinator`` and implement
|
|
``execute_task`` from scratch.
|
|
|
|
SubprocessCoordinator: implementing ``_build_execute_task_command``
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
:class:`~airflow.sdk.coordinators._subprocess.SubprocessCoordinator` handles
|
|
the full subprocess lifecycle. When a task is triggered, it:
|
|
|
|
1. Binds two ephemeral TCP server sockets on ``127.0.0.1``.
|
|
2. Spawns the subprocess with ``--comm=<host>:<port>`` and
|
|
``--logs=<host>:<port>`` appended to the command.
|
|
3. Waits for the subprocess to connect to both sockets.
|
|
4. Signal to subprocess to begin executing user code.
|
|
5. Forwards task log lines received over ``--logs`` to the Airflow log
|
|
infrastructure.
|
|
6. Tears everything down when the subprocess exits or the startup times out.
|
|
|
|
The ``--comm`` socket is the bidirectional task execution channel described in
|
|
`Language SDK (target language)`_. The ``--logs`` socket is used to forward
|
|
*infrastructure logs* (messages emitted by the SDK, not user code) to be merged
|
|
into Airflow worker logs.
|
|
|
|
Subclasses only need to supply the command to run and the wire-schema version
|
|
the subprocess understands:
|
|
|
|
.. code-block:: python
|
|
|
|
def _build_execute_task_command(self, *, what: TaskInstanceDTO) -> tuple[list[str], str]: ...
|
|
|
|
The method returns a ``(command, subprocess_schema_version)`` pair:
|
|
|
|
* ``command`` — the subprocess argv list. Do **not** include ``--comm`` or
|
|
``--logs``; the base class appends those flags after binding the sockets.
|
|
* ``subprocess_schema_version`` — the ``YYYY-MM-DD`` wire-schema version the
|
|
subprocess understands, used by the supervisor to negotiate message formats
|
|
across SDK versions. See `Supervisor Schema`_ below.
|
|
|
|
Call ``self._get_scan_roots()`` to retrieve the artifact directories the base
|
|
class has already resolved: the Dag bundle named by ``task_handler_bundle_name``,
|
|
or the task's own Dag bundle when it is unset, pinned for the whole task.
|
|
Subclasses should scan those directories rather than locating artifacts
|
|
themselves.
|
|
|
|
Supervisor Schema
|
|
~~~~~~~~~~~~~~~~~
|
|
|
|
The Supervisor Schema is the formal contract between the supervisor and a
|
|
lang-SDK subprocess. It is generated from Pydantic models defined in the
|
|
supervisor and published as a JSON Schema file at
|
|
``task-sdk/src/airflow/sdk/execution_time/schema/schema.json``. The file
|
|
describes every message type that can appear on the comm socket in either
|
|
direction, and carries an ``api_version`` field with a ``YYYY-MM-DD`` date
|
|
string that identifies the schema revision.
|
|
|
|
The schema evolves over time, so the SDK must tell the supervisor what API
|
|
version it was built against for this bridging to work. How the target-language
|
|
SDK uses ``schema.json`` to generate or validate its message types is covered in
|
|
`Language SDK (target language)`_ below.
|
|
|
|
Adding a new coordinator class
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
If you decide to implement a new coordinator, place it like this:
|
|
|
|
.. code-block:: text
|
|
|
|
task-sdk/src/airflow/sdk/coordinators/
|
|
<language>/
|
|
__init__.py
|
|
coordinator.py
|
|
|
|
Add unit tests under ``task-sdk/tests/`` following the same structure, and
|
|
integration tests in ``task-sdk/tests/integration/``.
|
|
|
|
|
|
Native Executable Bundle Format
|
|
-------------------------------
|
|
|
|
If the target runtime compiles to a self-contained native executable, the
|
|
:class:`~airflow.sdk.coordinators.executable.ExecutableCoordinator` can
|
|
discover and launch it automatically. For the coordinator to understand the
|
|
bundle correctly, extra metadata should be appended to the executable by a
|
|
custom bundling step at build-time.
|
|
|
|
See :ref:`Executable Bundle Spec` in Task SDK documentation for details.
|
|
|
|
|
|
Language SDK (target language)
|
|
-------------------------------
|
|
|
|
This section describes what the SDK in the target language must implement when
|
|
the coordinator uses the subprocess + TCP socket transport (i.e. when the
|
|
coordinator is a subclass of ``SubprocessCoordinator`` (this includes
|
|
``ExecutableCoordinator``). If a custom coordinator with a different transport
|
|
is used, the protocol between the coordinator and the SDK is entirely up to the
|
|
implementer.
|
|
|
|
Startup
|
|
~~~~~~~
|
|
|
|
The supervisor launches the subprocess with two command line arguments:
|
|
|
|
.. code-block:: text
|
|
|
|
--comm=<host>:<port>
|
|
--logs=<host>:<port>
|
|
|
|
The SDK process MUST be able to parse the arguments, and connect to both sockets
|
|
as soon as possible. The supervisor verifies that the connecting peer belongs
|
|
to the launched process tree, so the SDK MUST connect from the same process or
|
|
one of its descendants.
|
|
|
|
Once both connections are accepted, the supervisor sends a ``StartupDetails``
|
|
message on the comm socket to initiate execution.
|
|
|
|
Wire protocol
|
|
~~~~~~~~~~~~~
|
|
|
|
All communication on the ``--comm`` socket uses **length-prefixed MessagePack
|
|
frames**:
|
|
|
|
.. code-block:: text
|
|
|
|
[4-byte big-endian uint32: payload length][payload bytes]
|
|
|
|
The payload is a MessagePack-encoded array. Two shapes are used:
|
|
|
|
* SDK → Supervisor uses a 2-element array ``[id, body]``.
|
|
|
|
* ``id`` is an integer that identifies this request uniquely within the
|
|
connection. Responses echo the same ``id`` for correlation.
|
|
* ``body`` is the payload.
|
|
|
|
* Supervisor → SDK uses a 3-element array ``[id, body, error]``.
|
|
|
|
* ``id`` mirrors the request ``id`` if this message is a response to a request
|
|
initiated by the SDK .
|
|
* ``body`` is the payload, or ``null`` on error.
|
|
* ``error`` — an ``ErrorResponse`` map on failure, or ``null`` on success.
|
|
|
|
In both cases, the payload is a MessagePack map with a ``"type"`` key naming the
|
|
message. Maximum payload size is ``2³² - 1`` bytes. Sending a larger payload is
|
|
an error.
|
|
|
|
Message types
|
|
~~~~~~~~~~~~~
|
|
|
|
The complete field-level definition of every message type — names, types, and
|
|
constraints — lives in the Supervisor Schema file:
|
|
|
|
.. code-block:: text
|
|
|
|
task-sdk/src/airflow/sdk/execution_time/schema/schema.json
|
|
|
|
An SDK implementation can generate message types for the target language
|
|
directly from it with a code generator that supports JSON schema input.
|
|
|
|
Request/response correlation
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Each outbound request sent by the SDK carries a monotonically increasing integer
|
|
``id``. The supervisor echoes this ``id`` in the response frame so the SDK can
|
|
match responses to requests. The SDK MUST correlate by ``id`` whenever multiple
|
|
requests can be outstanding simultaneously (e.g. when tasks are executed
|
|
concurrently, or when a single task issues multiple requests).
|
|
|
|
Task subprocess lifecycle
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Before implementing logging, it helps to see the full lifecycle of the
|
|
subprocess relative to the ``--logs`` socket:
|
|
|
|
.. code-block:: text
|
|
|
|
1. Supervisor launches the subprocess with
|
|
--comm=<host>:<port> --logs=<host>:<port>
|
|
│
|
|
└─► 2. Subprocess starts. Its logger is already active, but the
|
|
--logs socket is not connected yet.
|
|
│
|
|
└─► 3. Subprocess connects to --comm and --logs.
|
|
│
|
|
└─► 4. Each subsequent record is sent over the
|
|
--logs socket.
|
|
|
|
The gap is stage 2: the SDK's own startup code (argument parsing, the connect
|
|
calls themselves) can already produce log records before the ``--logs`` socket
|
|
in stage 3 exists to carry them. See `Logging`_ below for how to handle that
|
|
gap.
|
|
|
|
Logging
|
|
~~~~~~~
|
|
|
|
The SDK should wire the native logging mechanism (the equivalent of Python's
|
|
``logging``, and/or whatever is popular in the ecosystem) to send logs emitted
|
|
during execution to Airflow's task log store, so they appear in the UI with the
|
|
rest of the output. Records travel over the ``--logs`` socket introduced in
|
|
`Startup`_.
|
|
|
|
Records produced during stage 2 of the `Task subprocess lifecycle`_ — before
|
|
the ``--logs`` socket connects — MUST NOT be dropped. Buffer them in memory and
|
|
flush the buffer, in order, as soon as the socket connects, before sending any
|
|
later record. Both the Go SDK and Java SDK already implement this buffer.
|
|
|
|
Log messages should be **newline-delimited JSON**. Each log is a UTF-8 encoded
|
|
JSON object in one line, terminated by a newline character (``\n``). Each object
|
|
is a structlog-style event:
|
|
|
|
.. code-block:: text
|
|
|
|
{"event": "Starting extraction", "level": "info", "logger": "com.example.SalesPipeline", "timestamp": "2026-06-22T12:00:00Z", "rows": 42}
|
|
|
|
* ``event`` is the log message.
|
|
* ``level`` is the level **name** in lower case. The following levels are
|
|
supported: ``critical``, ``error``, ``warning``, ``info``, ``debug``, and
|
|
``notset``. A line carrying any other level is dropped.
|
|
* ``timestamp`` is an RFC 3339 timestamp with a UTC offset, such as
|
|
``2026-06-22T12:00:00Z``. Airflow assumes that a timestamp without an offset
|
|
is in the time zone set by ``[core] default_timezone``, which is UTC by
|
|
default. If the worker's time zone differs from that setting, records without
|
|
an offset show the wrong time and appear out of order in the task log.
|
|
Include the seconds even when they are zero. The supervisor cannot parse a
|
|
timestamp without seconds, such as ``2026-06-22T12:00``. It moves that value
|
|
to an ``invalid_timestamp`` field and uses the time it received the record
|
|
instead.
|
|
* Any remaining keys are forwarded as structured fields on the log record.
|
|
|
|
See Python documentation on logging for details on log levels. Since log level
|
|
definitions on logging tools differ, the SDK should implement appropriate
|
|
translation logic to convert the levels.
|
|
|
|
An SDK SHOULD integrate with the language's established logging frameworks
|
|
rather than invent a new API, so task authors keep using the tools they already
|
|
know. The Java SDK, for example, ships adapters for many popular logging APIs,
|
|
including ``System.Logger`` and SLF4J.
|
|
|
|
The supervisor should not perform filtering on this socket; it should record
|
|
every line it receives. Instead, the supervisor should provide necessary
|
|
information for the SDK to perform filtering. ``SubprocessCoordinator`` (and
|
|
``ExecutableCoordinator`` since its a subclass) does this by setting the
|
|
following environment variables when launching the foreign runtime subprocess:
|
|
|
|
* ``AIRFLOW__LOGGING__LOGGING_LEVEL`` is the global threshold read configured in
|
|
``[logging] logging_level``, e.g. ``INFO``.
|
|
* ``AIRFLOW__LOGGING__NAMESPACE_LEVELS`` contains per-logger overrides from
|
|
``[logging] namespace_levels`` e.g. ``sqlalchemy=INFO, botocore=WARNING``.
|
|
|
|
See configuration documentation on the format and meaning of these
|
|
configurations.
|
|
|
|
The SDK MUST read these at startup and drop any record below the applicable
|
|
threshold *before* sending it, so that filtering matches what a Python task on
|
|
the same deployment would produce.
|
|
|
|
Error handling
|
|
~~~~~~~~~~~~~~
|
|
|
|
* If the task raises an unhandled error, the SDK MUST send a ``TaskState``
|
|
message with ``"state": "failed"`` before closing the comm socket.
|
|
* If the task fails but still has retries left, the SDK MUST send a
|
|
``RetryTask`` message instead so the supervisor moves the task to
|
|
``up_for_retry``. Field names MUST match the Supervisor Schema exactly — the
|
|
failure detail key is ``retry_reason``, not ``reason``.
|
|
* If the process exits without sending a terminal message, the supervisor marks
|
|
the task instance ``failed`` based on the abnormal exit, but the task log may
|
|
be incomplete.
|
|
* If the supervisor returns an ``ErrorResponse`` to a mid-task request, the SDK
|
|
SHOULD propagate it as an error to the task function.
|
|
|
|
|
|
Conformance
|
|
-----------
|
|
|
|
The sections above describe *how* to build an SDK. This section defines *what* an
|
|
SDK must do to be considered conformant: which TaskInstance states its runtime is
|
|
expected to report and which optional capabilities it may offer. Because SDKs mature
|
|
at different rates, they are not required to be identical; each one declares the
|
|
subset it actually supports.
|
|
|
|
The key words MUST, SHOULD, and MAY are used as described in :rfc:`2119`.
|
|
|
|
TaskInstance-state conformance
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
A running task reports its outcome to the supervisor as one of a small set of
|
|
state transitions, each carried by a specific message on the ``--comm`` socket
|
|
(see `Message types`_). Only the states a subprocess can *emit* are part of SDK
|
|
conformance; scheduler-owned states (``queued``, ``scheduled``, ``running``,
|
|
``restarting``, ``upstream_failed``) are set by Airflow and are never sent by the
|
|
runtime, so they are out of scope here.
|
|
|
|
.. list-table:: TaskInstance-state conformance
|
|
:header-rows: 1
|
|
:widths: 20 15 65
|
|
|
|
* - State
|
|
- Tier
|
|
- How it is reported / notes
|
|
* - ``success``
|
|
- MUST
|
|
- ``SucceedTask``, or a clean process exit (exit code 0).
|
|
* - ``failed``
|
|
- MUST
|
|
- ``TaskState`` with ``"state": "failed"``; also inferred from a non-zero
|
|
exit if no terminal message was sent.
|
|
* - ``up_for_retry``
|
|
- MUST
|
|
- ``RetryTask`` when the task fails but retries remain. The failure detail
|
|
key is ``retry_reason`` (not ``reason``).
|
|
* - ``skipped``
|
|
- SHOULD
|
|
- ``TaskState`` with ``"state": "skipped"``, for branching / skip semantics.
|
|
* - ``deferred``
|
|
- MAY
|
|
- ``DeferTask``. Requires the SDK to bridge to the triggerer.
|
|
* - ``up_for_reschedule``
|
|
- MAY
|
|
- ``RescheduleTask``, for reschedule-mode sensors.
|
|
* - ``awaiting_input``
|
|
- MAY
|
|
- ``AwaitInputTask``, for human-in-the-loop tasks.
|
|
* - ``removed``
|
|
- MAY
|
|
- ``TaskState`` with ``"state": "removed"``.
|
|
|
|
An SDK that only implements the MUST tier can already run ordinary tasks to
|
|
success or failure with retries; the SHOULD and MAY tiers unlock branching,
|
|
deferral, reschedule sensors, and human-in-the-loop.
|
|
|
|
Runtime capabilities
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Runtime capabilities describe what a task *body* can do while it executes, regardless of
|
|
whether the task was declared in a Python Dag (via ``@task.stub``) or in a Dag definition
|
|
authored in the target language. An SDK declares each one independently.
|
|
|
|
``task-logging`` (MUST)
|
|
The SDK forwards the task's log output — stdout / stderr and any structured log
|
|
records — to the supervisor over the ``--logs`` socket, so it surfaces in the task
|
|
log exactly like a Python task. A remote log *storage* (S3, GCS, …) is handled
|
|
uniformly by the supervisor and thus not part of SDK conformance.
|
|
|
|
``xcom-read-write`` (MUST)
|
|
The SDK exposes an API for the task to read and write XCom values across the Python
|
|
boundary.
|
|
|
|
``connection-read`` (MUST)
|
|
The task can resolve an Airflow Connection by id.
|
|
|
|
``variable-read-write`` (MUST)
|
|
The task can read, write, and delete Airflow Variables.
|
|
|
|
``self-contained-bundle`` (MUST)
|
|
The SDK's build artifact embeds its own Airflow metadata (``dag_id``, ``task_id`` and
|
|
the rest of the task descriptor) inside the *same* artifact as the task code, rather
|
|
than shipping it in a separate sidecar file, so the deployable unit is self-describing.
|
|
Each runtime satisfies this its own way — a Go binary carries an ``AFBNDL01`` metadata
|
|
trailer (see `Native Executable Bundle Format`_), a JVM artifact embeds it in the jar,
|
|
a Node bundle embeds it in the package.
|
|
|
|
``retry-policy`` (MAY)
|
|
The SDK lets task code inspect a failure and override whether the task retries or fails,
|
|
optionally with a custom retry delay, the equivalent of Python's ``RetryPolicy``. This is
|
|
distinct from reporting ``up_for_retry``: an SDK can support ordinary retries without
|
|
exposing a task-facing retry-policy API.
|
|
|
|
``task-state-store`` (MAY)
|
|
The task can read and write the per-task state store.
|
|
|
|
``asset-state-store`` (MAY)
|
|
The task can read and write the state store keyed by an asset name or URI. It says
|
|
nothing about whether the SDK can *declare* an asset, which is ``asset-definition``.
|
|
|
|
``asset-event-emit`` (MAY)
|
|
The task can emit asset events — outlet events, including alias events — as it runs.
|
|
|
|
``asset-event-read`` (MAY)
|
|
The task can read prior asset events.
|
|
|
|
``object-store`` (MAY)
|
|
The SDK exposes an object-storage API (an ``ObjectStoragePath`` equivalent) usable from
|
|
task code.
|
|
|
|
``listeners`` (MAY)
|
|
A task feeds Airflow's listener API, so a listener plugin sees it as it does a Python
|
|
task. OpenLineage is the main consumer: without this, a task of this SDK produces no
|
|
lineage facets of its own.
|
|
|
|
Dag definition
|
|
~~~~~~~~~~~~~~
|
|
|
|
A Dag definition lets an entire Dag — not just individual task bodies — be written in
|
|
the target language, without a Python Dag file. It is gated by a single umbrella
|
|
capability:
|
|
|
|
``dag-definition-authoring`` (SHOULD)
|
|
The SDK can define a whole Dag in the target language. The project intends every
|
|
Language SDK to reach this bar, but an SDK is not required to ship it before it is
|
|
ready; an SDK that only executes ``@task.stub`` tasks from a Python Dag is still useful
|
|
and remains conformant to the MUST tier.
|
|
|
|
The remaining Dag-definition capabilities are **conditional**: their tier applies only when
|
|
``dag-definition-authoring`` is supported (written ``†``). For an SDK that does not (yet) author Dags
|
|
in its own language they are *not applicable* (``n/a``) rather than unsupported.
|
|
|
|
``dag-args`` (MUST †)
|
|
Dag-level arguments (schedule, tags, default queue, and so on) can be set on a Dag definition.
|
|
|
|
``task-args`` (MUST †)
|
|
Task-level arguments (retries, pool, queue, and so on) can be set on a task of a Dag definition.
|
|
|
|
``dag-params`` (MUST †)
|
|
Dag-level params can be declared on a Dag definition.
|
|
|
|
``dag-callbacks`` (MAY †)
|
|
Dag-level callbacks (``on_failure_callback`` and the rest) can be declared on a Dag
|
|
definition as functions in the target language, and Airflow runs them. This is a
|
|
Dag-definition concern only: the callbacks of a Python Dag are Python functions that the
|
|
Python runtime runs whichever SDK executes the tasks.
|
|
|
|
``explicit-dependencies`` (MUST †)
|
|
Order-only dependencies can be drawn between tasks, the equivalent of Python's ``>>`` and
|
|
``<<``.
|
|
|
|
``edge-labels`` (MAY †)
|
|
A dependency can carry a label, the equivalent of Python's
|
|
``loaded >> Label("when empty") >> notify``, and the Dag graph shows it.
|
|
|
|
``taskflow-dependencies`` (MUST †)
|
|
Task dependencies can be expressed in a TaskFlow style, including passing a task's
|
|
return value to a downstream task.
|
|
|
|
``task-group`` (MAY †)
|
|
Tasks can be organised into task groups.
|
|
|
|
``task-loops`` (MAY †)
|
|
A task group can be repeated at runtime, the equivalent of Python's ``@task_group``
|
|
with ``.loop()``, bounded by ``max_iterations`` and an optional ``until`` condition.
|
|
|
|
``if-else-branching`` (SHOULD †)
|
|
A task can decide between two downstream paths and skip the one not taken.
|
|
|
|
``switch-case-branching`` (SHOULD †)
|
|
A task can pick one of several downstream paths and skip the others.
|
|
|
|
``trigger-dag-run`` (MAY †)
|
|
A Dag definition can declare a task that triggers a run of another Dag, like
|
|
``TriggerDagRunOperator``.
|
|
|
|
``dynamic-dag-generation`` (MAY †)
|
|
One artifact can generate several Dags from code, for example in a loop.
|
|
|
|
``human-in-the-loop`` (MAY †)
|
|
A Dag definition can declare a task that waits for human input.
|
|
|
|
``dag-test`` (SHOULD †)
|
|
A Dag definition can be exercised locally with ``airflow dags test``.
|
|
|
|
``asset-definition`` (MAY †)
|
|
Assets can be declared in the target language: the equivalents of Python's ``Asset``,
|
|
``AssetAlias``, ``AssetAll`` / ``AssetAny``, and the partition mappers a partitioned
|
|
asset timetable consumes.
|
|
|
|
``asset-scheduling`` (MAY †)
|
|
A Dag definition can be scheduled on asset events, including the ``AssetWatcher``
|
|
equivalent that turns an external trigger into those events.
|
|
|
|
Python Dag with stub TaskHandler
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
These capabilities cover tasks declared in a Python Dag with the ``@task.stub`` decorator and
|
|
implemented in the target language.
|
|
|
|
``mixed-lang-stub-target`` (MUST)
|
|
The SDK can execute a task declared in a Python Dag with the ``@task.stub``
|
|
decorator. This is the primary execution path for every Language SDK.
|
|
|
|
``taskflow-binding`` (MUST)
|
|
Arguments bound to a ``@task.stub`` target in the Python TaskFlow Dag — literal values
|
|
and upstream XCom results alike — are delivered to the task function's parameters at
|
|
runtime.
|
|
|
|
Shared capabilities
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
These capabilities exist in both authoring modes, and an SDK may support them in one mode but not
|
|
the other, so each one is declared once per mode. In a Dag definition the tier applies only when
|
|
``dag-definition-authoring`` is supported.
|
|
|
|
``mapped-tasks`` (MAY)
|
|
A task can be expanded at runtime via dynamic task mapping, and each mapped instance
|
|
receives its own arguments.
|
|
|
|
``asset-inlets-outlets`` (MAY)
|
|
A task's asset inlets and outlets are honored, so a successful task emits events for its
|
|
outlets.
|
|
|
|
``task-callbacks`` (MAY)
|
|
Task-level callbacks (``on_failure_callback`` and the rest) run for a task of this SDK.
|
|
The two authoring modes ask different things of the SDK: in a Dag definition the SDK
|
|
declares the callback as a function in its own language, while for a ``@task.stub`` task
|
|
the callback is declared in the Python Dag and the SDK's execution path has to get it run.
|
|
|
|
Compatibility matrix
|
|
~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The dimensions above are prose; each SDK also declares them *machine-readably* in one
|
|
hand-authored ``<sdk>/capabilities.yaml``, from which prek hooks generate the published tables:
|
|
|
|
.. code-block:: text
|
|
|
|
java-sdk/capabilities.yaml <- the only file you edit
|
|
|
|
|
+-- hook: update-java-sdk-readme-matrix
|
|
| +--> java-sdk/README.md (user-facing landing page)
|
|
| +--> java-sdk/sdk/module.md (Dokka -> the published API reference)
|
|
|
|
|
+-- hook: update-lang-sdk-central-matrix (reads every SDK in LANG_SDKS)
|
|
+--> airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst
|
|
(consolidated cross-SDK matrix)
|
|
|
|
Each hook rewrites its targets and exits non-zero when a target was stale, so a drifted table fails
|
|
the build. Keep the manifest out of whatever the SDK publishes — it describes the SDK rather than
|
|
being part of it; for Java that means sitting above every subproject in ``settings.gradle.kts``.
|
|
|
|
A supported dimension may also record ``sdk_version``, the SDK's own release that shipped it,
|
|
alongside ``since`` (the Airflow version). It is optional: name the release that ships, or will
|
|
ship, the dimension, and omit it when that is not settled yet. A shared capability holds one
|
|
entry per authoring mode:
|
|
|
|
.. code-block:: yaml
|
|
|
|
mapped-tasks:
|
|
dag-definition:
|
|
supported: false
|
|
stub-task-handler:
|
|
supported: true
|
|
since: "3.4.0"
|
|
sdk_version: "1.0.0"
|
|
|
|
To add an SDK, register it in ``LANG_SDKS`` (its ``id``, ``display_name`` and file locations) in
|
|
``scripts/ci/prek/lang_sdk_compat_matrix.py``, write
|
|
a ``capabilities.yaml`` in the same schema, and add the equivalent hook. Adding or renaming a
|
|
dimension means editing ``STATE_DIMENSIONS`` / ``CAPABILITY_DIMENSIONS`` there **and** the prose
|
|
above in the same PR — the renderer validates every manifest against that list.
|
|
|
|
|
|
Testing
|
|
-------
|
|
|
|
A language SDK implementation should include:
|
|
|
|
* **Unit tests** for the framing layer (encode/decode round-trips, oversized
|
|
frames, corrupt length prefixes).
|
|
* **Unit tests** for each message type in both directions.
|
|
* **Integration tests** against a live supervisor using Breeze.
|