Files
airflow/contributing-docs/30_new_language_sdk.rst
T
Jason(Zhe-You) Liu 719a685079 Add consolidated Language SDK compatibility matrix (#71757)
* Add consolidated Language SDK compatibility matrix

Users comparing Language SDKs had to read each SDK's README, and the
per-SDK manifests lagged behind what the SDKs actually ship. One generated
matrix, split by authoring mode, shows what each SDK supports for a Dag
definition and for a Python Dag with stub TaskHandler.

* Add callbacks, listeners and provider operators to the SDK matrix

Users comparing Language SDKs could not see where callbacks, Airflow's
listener API and Python provider operators stand, because the matrix had
no dimension for any of them. Callbacks ask different things of an SDK per
authoring mode, so they are declared per mode like the other shared
capabilities.

* Address review on the Language SDK matrix generator

The umbrella capability and the authoring mode both read `dag-definition`,
which gives one name two meanings in the same manifest. A row's kind also
belongs in its entry key: state and capability names happen not to collide
today, so the lookup worked by accident rather than by design.

* Rename the dynamic-task-mapping dimension to mapped-tasks

The dimension is about whether a task can be a mapped task, which the
shorter name says directly and keeps the matrix row readable next to the
other task-level dimensions.

* Split callbacks into Dag-level and task-level dimensions

The callbacks of a Python Dag run whichever SDK executes its stub tasks, so
``dag-callbacks`` is a Dag-definition concern only, while ``task-callbacks``
stays declared per authoring mode. Drop ``provider-operators`` until ADR-0009
is accepted, and reuse the shared ``capabilities_doc`` fixture in the per-SDK
README tests.

* Fix the hook attribution in the matrix generation diagram

``index.rst`` is written by ``update-lang-sdk-central-matrix`` from every SDK
in ``LANG_SDKS``, not by the per-SDK README hook.

---------
2026-10-23 22:28:33 +08:00

642 lines
27 KiB
ReStructuredText

.. Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
.. http://www.apache.org/licenses/LICENSE-2.0
.. Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
Creating a new Language SDK
===========================
Starting from 3.3, the standard Airflow workers can run task code implemented in
languages other than Python, using a foreign Language SDK. This document
describes how a new Language SDK can be contributed, so Airflow can execute
tasks implemented in the language. It is also the normative *conformance
specification* a Language SDK is expected to meet: the `Conformance`_ section
defines the exact TaskInstance states and capabilities an SDK MUST, SHOULD, or
MAY support.
Two components are needed for Airflow to understand how to execute such a task:
* A **coordinator implemented in Python** for Airflow to understand how to wire
up the foreign package.
* A **language SDK implemented in the target language** to talk to the
coordinator.
The two components are largely independent: the coordinator decides *how* to
start the foreign runtime and *how* to communicate with it, and the language SDK
implements the other end of whatever protocol the coordinator chooses. There is
no single mandated communication mechanism.
For a diagram-driven overview of how a task runs across the Python and Coordinator
paths, see `Task execution architecture <31_task_execution_architecture.rst>`__.
.. contents:: Table of Contents
:depth: 2
:local:
Python Coordinator
------------------
The coordinator is a Python class that Airflow calls when a stub task with a
matching queue is triggered. Its sole required method is ``execute_task``, which
must start the foreign runtime, hand off the task, and return the final task
state. The base class is
:class:`airflow.sdk.execution_time.coordinator.BaseCoordinator`.
A coordinator can communicate with the foreign runtime in any way it chooses.
This can be a subprocess over TCP sockets, a gRPC server, shared memory, a
message queue, or anything else. The choice of transport is entirely up to the
coordinator and its SDK counterpart.
Choosing an implementation path
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In practice, almost all coordinators will fall into one of the following
categories. Choose the one that fits the target runtime:
Use :class:`~airflow.sdk.coordinators.executable.ExecutableCoordinator`
If the language SDK compiles to a self-contained native executable, the
existing ``ExecutableCoordinator`` can discover and launch it with zero
Python code. The executable just needs to carry an ``AFBNDL01`` metadata
trailer (see `Native Executable Bundle Format`_ below). This is the simplest
path since you would not need to write any Python code.
Subclass :class:`~airflow.sdk.coordinators._subprocess.SubprocessCoordinator`
If a command is needed (e.g. `java` for JRE, `node` for Node), but
the produced runtime can communicate over TCP, consider subclassing
``SubprocessCoordinator`` and implement ``_build_execute_task_command``.
The base class handles all TCP socket lifecycle, process ownership
verification, startup draining, and teardown.
Subclass ``BaseCoordinator`` directly
For runtimes that use a completely different transport (gRPC, shared memory,
a persistent daemon, etc.), subclass ``BaseCoordinator`` and implement
``execute_task`` from scratch.
SubprocessCoordinator: implementing ``_build_execute_task_command``
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
:class:`~airflow.sdk.coordinators._subprocess.SubprocessCoordinator` handles
the full subprocess lifecycle. When a task is triggered, it:
1. Binds two ephemeral TCP server sockets on ``127.0.0.1``.
2. Spawns the subprocess with ``--comm=<host>:<port>`` and
``--logs=<host>:<port>`` appended to the command.
3. Waits for the subprocess to connect to both sockets.
4. Signal to subprocess to begin executing user code.
5. Forwards task log lines received over ``--logs`` to the Airflow log
infrastructure.
6. Tears everything down when the subprocess exits or the startup times out.
The ``--comm`` socket is the bidirectional task execution channel described in
`Language SDK (target language)`_. The ``--logs`` socket is used to forward
*infrastructure logs* (messages emitted by the SDK, not user code) to be merged
into Airflow worker logs.
Subclasses only need to supply the command to run and the wire-schema version
the subprocess understands:
.. code-block:: python
def _build_execute_task_command(self, *, what: TaskInstanceDTO) -> tuple[list[str], str]: ...
The method returns a ``(command, subprocess_schema_version)`` pair:
* ``command`` — the subprocess argv list. Do **not** include ``--comm`` or
``--logs``; the base class appends those flags after binding the sockets.
* ``subprocess_schema_version`` — the ``YYYY-MM-DD`` wire-schema version the
subprocess understands, used by the supervisor to negotiate message formats
across SDK versions. See `Supervisor Schema`_ below.
Call ``self._get_scan_roots()`` to retrieve the artifact directories the base
class has already resolved: the Dag bundle named by ``task_handler_bundle_name``,
or the task's own Dag bundle when it is unset, pinned for the whole task.
Subclasses should scan those directories rather than locating artifacts
themselves.
Supervisor Schema
~~~~~~~~~~~~~~~~~
The Supervisor Schema is the formal contract between the supervisor and a
lang-SDK subprocess. It is generated from Pydantic models defined in the
supervisor and published as a JSON Schema file at
``task-sdk/src/airflow/sdk/execution_time/schema/schema.json``. The file
describes every message type that can appear on the comm socket in either
direction, and carries an ``api_version`` field with a ``YYYY-MM-DD`` date
string that identifies the schema revision.
The schema evolves over time, so the SDK must tell the supervisor what API
version it was built against for this bridging to work. How the target-language
SDK uses ``schema.json`` to generate or validate its message types is covered in
`Language SDK (target language)`_ below.
Adding a new coordinator class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you decide to implement a new coordinator, place it like this:
.. code-block:: text
task-sdk/src/airflow/sdk/coordinators/
<language>/
__init__.py
coordinator.py
Add unit tests under ``task-sdk/tests/`` following the same structure, and
integration tests in ``task-sdk/tests/integration/``.
Native Executable Bundle Format
-------------------------------
If the target runtime compiles to a self-contained native executable, the
:class:`~airflow.sdk.coordinators.executable.ExecutableCoordinator` can
discover and launch it automatically. For the coordinator to understand the
bundle correctly, extra metadata should be appended to the executable by a
custom bundling step at build-time.
See :ref:`Executable Bundle Spec` in Task SDK documentation for details.
Language SDK (target language)
-------------------------------
This section describes what the SDK in the target language must implement when
the coordinator uses the subprocess + TCP socket transport (i.e. when the
coordinator is a subclass of ``SubprocessCoordinator`` (this includes
``ExecutableCoordinator``). If a custom coordinator with a different transport
is used, the protocol between the coordinator and the SDK is entirely up to the
implementer.
Startup
~~~~~~~
The supervisor launches the subprocess with two command line arguments:
.. code-block:: text
--comm=<host>:<port>
--logs=<host>:<port>
The SDK process MUST be able to parse the arguments, and connect to both sockets
as soon as possible. The supervisor verifies that the connecting peer belongs
to the launched process tree, so the SDK MUST connect from the same process or
one of its descendants.
Once both connections are accepted, the supervisor sends a ``StartupDetails``
message on the comm socket to initiate execution.
Wire protocol
~~~~~~~~~~~~~
All communication on the ``--comm`` socket uses **length-prefixed MessagePack
frames**:
.. code-block:: text
[4-byte big-endian uint32: payload length][payload bytes]
The payload is a MessagePack-encoded array. Two shapes are used:
* SDK → Supervisor uses a 2-element array ``[id, body]``.
* ``id`` is an integer that identifies this request uniquely within the
connection. Responses echo the same ``id`` for correlation.
* ``body`` is the payload.
* Supervisor → SDK uses a 3-element array ``[id, body, error]``.
* ``id`` mirrors the request ``id`` if this message is a response to a request
initiated by the SDK .
* ``body`` is the payload, or ``null`` on error.
* ``error`` — an ``ErrorResponse`` map on failure, or ``null`` on success.
In both cases, the payload is a MessagePack map with a ``"type"`` key naming the
message. Maximum payload size is ``2³² - 1`` bytes. Sending a larger payload is
an error.
Message types
~~~~~~~~~~~~~
The complete field-level definition of every message type — names, types, and
constraints — lives in the Supervisor Schema file:
.. code-block:: text
task-sdk/src/airflow/sdk/execution_time/schema/schema.json
An SDK implementation can generate message types for the target language
directly from it with a code generator that supports JSON schema input.
Request/response correlation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Each outbound request sent by the SDK carries a monotonically increasing integer
``id``. The supervisor echoes this ``id`` in the response frame so the SDK can
match responses to requests. The SDK MUST correlate by ``id`` whenever multiple
requests can be outstanding simultaneously (e.g. when tasks are executed
concurrently, or when a single task issues multiple requests).
Task subprocess lifecycle
~~~~~~~~~~~~~~~~~~~~~~~~~
Before implementing logging, it helps to see the full lifecycle of the
subprocess relative to the ``--logs`` socket:
.. code-block:: text
1. Supervisor launches the subprocess with
--comm=<host>:<port> --logs=<host>:<port>
│
└─► 2. Subprocess starts. Its logger is already active, but the
--logs socket is not connected yet.
│
└─► 3. Subprocess connects to --comm and --logs.
│
└─► 4. Each subsequent record is sent over the
--logs socket.
The gap is stage 2: the SDK's own startup code (argument parsing, the connect
calls themselves) can already produce log records before the ``--logs`` socket
in stage 3 exists to carry them. See `Logging`_ below for how to handle that
gap.
Logging
~~~~~~~
The SDK should wire the native logging mechanism (the equivalent of Python's
``logging``, and/or whatever is popular in the ecosystem) to send logs emitted
during execution to Airflow's task log store, so they appear in the UI with the
rest of the output. Records travel over the ``--logs`` socket introduced in
`Startup`_.
Records produced during stage 2 of the `Task subprocess lifecycle`_ — before
the ``--logs`` socket connects — MUST NOT be dropped. Buffer them in memory and
flush the buffer, in order, as soon as the socket connects, before sending any
later record. Both the Go SDK and Java SDK already implement this buffer.
Log messages should be **newline-delimited JSON**. Each log is a UTF-8 encoded
JSON object in one line, terminated by a newline character (``\n``). Each object
is a structlog-style event:
.. code-block:: text
{"event": "Starting extraction", "level": "info", "logger": "com.example.SalesPipeline", "timestamp": "2026-06-22T12:00:00Z", "rows": 42}
* ``event`` is the log message.
* ``level`` is the level **name** in lower case. The following levels are
supported: ``critical``, ``error``, ``warning``, ``info``, ``debug``, and
``notset``. A line carrying any other level is dropped.
* ``timestamp`` is an RFC 3339 timestamp with a UTC offset, such as
``2026-06-22T12:00:00Z``. Airflow assumes that a timestamp without an offset
is in the time zone set by ``[core] default_timezone``, which is UTC by
default. If the worker's time zone differs from that setting, records without
an offset show the wrong time and appear out of order in the task log.
Include the seconds even when they are zero. The supervisor cannot parse a
timestamp without seconds, such as ``2026-06-22T12:00``. It moves that value
to an ``invalid_timestamp`` field and uses the time it received the record
instead.
* Any remaining keys are forwarded as structured fields on the log record.
See Python documentation on logging for details on log levels. Since log level
definitions on logging tools differ, the SDK should implement appropriate
translation logic to convert the levels.
An SDK SHOULD integrate with the language's established logging frameworks
rather than invent a new API, so task authors keep using the tools they already
know. The Java SDK, for example, ships adapters for many popular logging APIs,
including ``System.Logger`` and SLF4J.
The supervisor should not perform filtering on this socket; it should record
every line it receives. Instead, the supervisor should provide necessary
information for the SDK to perform filtering. ``SubprocessCoordinator`` (and
``ExecutableCoordinator`` since its a subclass) does this by setting the
following environment variables when launching the foreign runtime subprocess:
* ``AIRFLOW__LOGGING__LOGGING_LEVEL`` is the global threshold read configured in
``[logging] logging_level``, e.g. ``INFO``.
* ``AIRFLOW__LOGGING__NAMESPACE_LEVELS`` contains per-logger overrides from
``[logging] namespace_levels`` e.g. ``sqlalchemy=INFO, botocore=WARNING``.
See configuration documentation on the format and meaning of these
configurations.
The SDK MUST read these at startup and drop any record below the applicable
threshold *before* sending it, so that filtering matches what a Python task on
the same deployment would produce.
Error handling
~~~~~~~~~~~~~~
* If the task raises an unhandled error, the SDK MUST send a ``TaskState``
message with ``"state": "failed"`` before closing the comm socket.
* If the task fails but still has retries left, the SDK MUST send a
``RetryTask`` message instead so the supervisor moves the task to
``up_for_retry``. Field names MUST match the Supervisor Schema exactly — the
failure detail key is ``retry_reason``, not ``reason``.
* If the process exits without sending a terminal message, the supervisor marks
the task instance ``failed`` based on the abnormal exit, but the task log may
be incomplete.
* If the supervisor returns an ``ErrorResponse`` to a mid-task request, the SDK
SHOULD propagate it as an error to the task function.
Conformance
-----------
The sections above describe *how* to build an SDK. This section defines *what* an
SDK must do to be considered conformant: which TaskInstance states its runtime is
expected to report and which optional capabilities it may offer. Because SDKs mature
at different rates, they are not required to be identical; each one declares the
subset it actually supports.
The key words MUST, SHOULD, and MAY are used as described in :rfc:`2119`.
TaskInstance-state conformance
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A running task reports its outcome to the supervisor as one of a small set of
state transitions, each carried by a specific message on the ``--comm`` socket
(see `Message types`_). Only the states a subprocess can *emit* are part of SDK
conformance; scheduler-owned states (``queued``, ``scheduled``, ``running``,
``restarting``, ``upstream_failed``) are set by Airflow and are never sent by the
runtime, so they are out of scope here.
.. list-table:: TaskInstance-state conformance
:header-rows: 1
:widths: 20 15 65
* - State
- Tier
- How it is reported / notes
* - ``success``
- MUST
- ``SucceedTask``, or a clean process exit (exit code 0).
* - ``failed``
- MUST
- ``TaskState`` with ``"state": "failed"``; also inferred from a non-zero
exit if no terminal message was sent.
* - ``up_for_retry``
- MUST
- ``RetryTask`` when the task fails but retries remain. The failure detail
key is ``retry_reason`` (not ``reason``).
* - ``skipped``
- SHOULD
- ``TaskState`` with ``"state": "skipped"``, for branching / skip semantics.
* - ``deferred``
- MAY
- ``DeferTask``. Requires the SDK to bridge to the triggerer.
* - ``up_for_reschedule``
- MAY
- ``RescheduleTask``, for reschedule-mode sensors.
* - ``awaiting_input``
- MAY
- ``AwaitInputTask``, for human-in-the-loop tasks.
* - ``removed``
- MAY
- ``TaskState`` with ``"state": "removed"``.
An SDK that only implements the MUST tier can already run ordinary tasks to
success or failure with retries; the SHOULD and MAY tiers unlock branching,
deferral, reschedule sensors, and human-in-the-loop.
Runtime capabilities
~~~~~~~~~~~~~~~~~~~~~~
Runtime capabilities describe what a task *body* can do while it executes, regardless of
whether the task was declared in a Python Dag (via ``@task.stub``) or in a Dag definition
authored in the target language. An SDK declares each one independently.
``task-logging`` (MUST)
The SDK forwards the task's log output — stdout / stderr and any structured log
records — to the supervisor over the ``--logs`` socket, so it surfaces in the task
log exactly like a Python task. A remote log *storage* (S3, GCS, …) is handled
uniformly by the supervisor and thus not part of SDK conformance.
``xcom-read-write`` (MUST)
The SDK exposes an API for the task to read and write XCom values across the Python
boundary.
``connection-read`` (MUST)
The task can resolve an Airflow Connection by id.
``variable-read-write`` (MUST)
The task can read, write, and delete Airflow Variables.
``self-contained-bundle`` (MUST)
The SDK's build artifact embeds its own Airflow metadata (``dag_id``, ``task_id`` and
the rest of the task descriptor) inside the *same* artifact as the task code, rather
than shipping it in a separate sidecar file, so the deployable unit is self-describing.
Each runtime satisfies this its own way — a Go binary carries an ``AFBNDL01`` metadata
trailer (see `Native Executable Bundle Format`_), a JVM artifact embeds it in the jar,
a Node bundle embeds it in the package.
``retry-policy`` (MAY)
The SDK lets task code inspect a failure and override whether the task retries or fails,
optionally with a custom retry delay, the equivalent of Python's ``RetryPolicy``. This is
distinct from reporting ``up_for_retry``: an SDK can support ordinary retries without
exposing a task-facing retry-policy API.
``task-state-store`` (MAY)
The task can read and write the per-task state store.
``asset-state-store`` (MAY)
The task can read and write the state store keyed by an asset name or URI. It says
nothing about whether the SDK can *declare* an asset, which is ``asset-definition``.
``asset-event-emit`` (MAY)
The task can emit asset events — outlet events, including alias events — as it runs.
``asset-event-read`` (MAY)
The task can read prior asset events.
``object-store`` (MAY)
The SDK exposes an object-storage API (an ``ObjectStoragePath`` equivalent) usable from
task code.
``listeners`` (MAY)
A task feeds Airflow's listener API, so a listener plugin sees it as it does a Python
task. OpenLineage is the main consumer: without this, a task of this SDK produces no
lineage facets of its own.
Dag definition
~~~~~~~~~~~~~~
A Dag definition lets an entire Dag — not just individual task bodies — be written in
the target language, without a Python Dag file. It is gated by a single umbrella
capability:
``dag-definition-authoring`` (SHOULD)
The SDK can define a whole Dag in the target language. The project intends every
Language SDK to reach this bar, but an SDK is not required to ship it before it is
ready; an SDK that only executes ``@task.stub`` tasks from a Python Dag is still useful
and remains conformant to the MUST tier.
The remaining Dag-definition capabilities are **conditional**: their tier applies only when
``dag-definition-authoring`` is supported (written ``†``). For an SDK that does not (yet) author Dags
in its own language they are *not applicable* (``n/a``) rather than unsupported.
``dag-args`` (MUST †)
Dag-level arguments (schedule, tags, default queue, and so on) can be set on a Dag definition.
``task-args`` (MUST †)
Task-level arguments (retries, pool, queue, and so on) can be set on a task of a Dag definition.
``dag-params`` (MUST †)
Dag-level params can be declared on a Dag definition.
``dag-callbacks`` (MAY †)
Dag-level callbacks (``on_failure_callback`` and the rest) can be declared on a Dag
definition as functions in the target language, and Airflow runs them. This is a
Dag-definition concern only: the callbacks of a Python Dag are Python functions that the
Python runtime runs whichever SDK executes the tasks.
``explicit-dependencies`` (MUST †)
Order-only dependencies can be drawn between tasks, the equivalent of Python's ``>>`` and
``<<``.
``edge-labels`` (MAY †)
A dependency can carry a label, the equivalent of Python's
``loaded >> Label("when empty") >> notify``, and the Dag graph shows it.
``taskflow-dependencies`` (MUST †)
Task dependencies can be expressed in a TaskFlow style, including passing a task's
return value to a downstream task.
``task-group`` (MAY †)
Tasks can be organised into task groups.
``task-loops`` (MAY †)
A task group can be repeated at runtime, the equivalent of Python's ``@task_group``
with ``.loop()``, bounded by ``max_iterations`` and an optional ``until`` condition.
``if-else-branching`` (SHOULD †)
A task can decide between two downstream paths and skip the one not taken.
``switch-case-branching`` (SHOULD †)
A task can pick one of several downstream paths and skip the others.
``trigger-dag-run`` (MAY †)
A Dag definition can declare a task that triggers a run of another Dag, like
``TriggerDagRunOperator``.
``dynamic-dag-generation`` (MAY †)
One artifact can generate several Dags from code, for example in a loop.
``human-in-the-loop`` (MAY †)
A Dag definition can declare a task that waits for human input.
``dag-test`` (SHOULD †)
A Dag definition can be exercised locally with ``airflow dags test``.
``asset-definition`` (MAY †)
Assets can be declared in the target language: the equivalents of Python's ``Asset``,
``AssetAlias``, ``AssetAll`` / ``AssetAny``, and the partition mappers a partitioned
asset timetable consumes.
``asset-scheduling`` (MAY †)
A Dag definition can be scheduled on asset events, including the ``AssetWatcher``
equivalent that turns an external trigger into those events.
Python Dag with stub TaskHandler
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
These capabilities cover tasks declared in a Python Dag with the ``@task.stub`` decorator and
implemented in the target language.
``mixed-lang-stub-target`` (MUST)
The SDK can execute a task declared in a Python Dag with the ``@task.stub``
decorator. This is the primary execution path for every Language SDK.
``taskflow-binding`` (MUST)
Arguments bound to a ``@task.stub`` target in the Python TaskFlow Dag — literal values
and upstream XCom results alike — are delivered to the task function's parameters at
runtime.
Shared capabilities
~~~~~~~~~~~~~~~~~~~
These capabilities exist in both authoring modes, and an SDK may support them in one mode but not
the other, so each one is declared once per mode. In a Dag definition the tier applies only when
``dag-definition-authoring`` is supported.
``mapped-tasks`` (MAY)
A task can be expanded at runtime via dynamic task mapping, and each mapped instance
receives its own arguments.
``asset-inlets-outlets`` (MAY)
A task's asset inlets and outlets are honored, so a successful task emits events for its
outlets.
``task-callbacks`` (MAY)
Task-level callbacks (``on_failure_callback`` and the rest) run for a task of this SDK.
The two authoring modes ask different things of the SDK: in a Dag definition the SDK
declares the callback as a function in its own language, while for a ``@task.stub`` task
the callback is declared in the Python Dag and the SDK's execution path has to get it run.
Compatibility matrix
~~~~~~~~~~~~~~~~~~~~
The dimensions above are prose; each SDK also declares them *machine-readably* in one
hand-authored ``<sdk>/capabilities.yaml``, from which prek hooks generate the published tables:
.. code-block:: text
java-sdk/capabilities.yaml <- the only file you edit
|
+-- hook: update-java-sdk-readme-matrix
| +--> java-sdk/README.md (user-facing landing page)
| +--> java-sdk/sdk/module.md (Dokka -> the published API reference)
|
+-- hook: update-lang-sdk-central-matrix (reads every SDK in LANG_SDKS)
+--> airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst
(consolidated cross-SDK matrix)
Each hook rewrites its targets and exits non-zero when a target was stale, so a drifted table fails
the build. Keep the manifest out of whatever the SDK publishes — it describes the SDK rather than
being part of it; for Java that means sitting above every subproject in ``settings.gradle.kts``.
A supported dimension may also record ``sdk_version``, the SDK's own release that shipped it,
alongside ``since`` (the Airflow version). It is optional: name the release that ships, or will
ship, the dimension, and omit it when that is not settled yet. A shared capability holds one
entry per authoring mode:
.. code-block:: yaml
mapped-tasks:
dag-definition:
supported: false
stub-task-handler:
supported: true
since: "3.4.0"
sdk_version: "1.0.0"
To add an SDK, register it in ``LANG_SDKS`` (its ``id``, ``display_name`` and file locations) in
``scripts/ci/prek/lang_sdk_compat_matrix.py``, write
a ``capabilities.yaml`` in the same schema, and add the equivalent hook. Adding or renaming a
dimension means editing ``STATE_DIMENSIONS`` / ``CAPABILITY_DIMENSIONS`` there **and** the prose
above in the same PR — the renderer validates every manifest against that list.
Testing
-------
A language SDK implementation should include:
* **Unit tests** for the framing layer (encode/decode round-trips, oversized
frames, corrupt length prefixes).
* **Unit tests** for each message type in both directions.
* **Integration tests** against a live supervisor using Breeze.