Files
Jason(Zhe-You) Liu 719a685079 Add consolidated Language SDK compatibility matrix (#71757)
* Add consolidated Language SDK compatibility matrix

Users comparing Language SDKs had to read each SDK's README, and the
per-SDK manifests lagged behind what the SDKs actually ship. One generated
matrix, split by authoring mode, shows what each SDK supports for a Dag
definition and for a Python Dag with stub TaskHandler.

* Add callbacks, listeners and provider operators to the SDK matrix

Users comparing Language SDKs could not see where callbacks, Airflow's
listener API and Python provider operators stand, because the matrix had
no dimension for any of them. Callbacks ask different things of an SDK per
authoring mode, so they are declared per mode like the other shared
capabilities.

* Address review on the Language SDK matrix generator

The umbrella capability and the authoring mode both read `dag-definition`,
which gives one name two meanings in the same manifest. A row's kind also
belongs in its entry key: state and capability names happen not to collide
today, so the lookup worked by accident rather than by design.

* Rename the dynamic-task-mapping dimension to mapped-tasks

The dimension is about whether a task can be a mapped task, which the
shorter name says directly and keeps the matrix row readable next to the
other task-level dimensions.

* Split callbacks into Dag-level and task-level dimensions

The callbacks of a Python Dag run whichever SDK executes its stub tasks, so
``dag-callbacks`` is a Dag-definition concern only, while ``task-callbacks``
stays declared per authoring mode. Drop ``provider-operators`` until ADR-0009
is accepted, and reuse the shared ``capabilities_doc`` fixture in the per-SDK
README tests.

* Fix the hook attribution in the matrix generation diagram

``index.rst`` is written by ``update-lang-sdk-central-matrix`` from every SDK
in ``LANG_SDKS``, not by the per-SDK README hook.

---------
2026-10-23 22:28:33 +08:00

11 KiB
Raw Permalink Blame History

Airflow Java SDK

Write Apache Airflow Dags and tasks in Java, Kotlin, Scala or any other JVM language.

Status: Beta · Java 11+

Installation

Apply the Gradle plugin, and add the SDK and its annotation processor:

plugins {
    id("org.apache.airflow.sdk") version "<version>"
}

dependencies {
    annotationProcessor("org.apache.airflow:airflow-sdk-processor:<version>")
    implementation("org.apache.airflow:airflow-sdk:<version>")
}

airflowBundle {
    mainClass = "com.mycompany.airflow.hello.Main"
}

Replace <version> with a release from Maven Central. See the guide for Maven.

Quick start

Declare a Dag:

package com.mycompany.airflow.hello;

import org.apache.airflow.sdk.*;

@Airflow.Dag(id = "java_hello", schedule = "@daily", queue = "java")
public class HelloDag {

  @Airflow.Task
  public long extract() {
    return 42L;
  }

  @Airflow.Task
  public void report(long rows) {
    System.out.println("extracted " + rows + " rows");
  }

  @Airflow.Deps
  static class Wiring implements HelloDagDeps {
    void depends() {
      report(extract());
    }
  }
}

Serve it from the main class:

package com.mycompany.airflow.hello;

import org.apache.airflow.sdk.Bundle;
import org.apache.airflow.sdk.Server;

public class Main {
  public static void main(String[] args) {
    Server.create(args).serve(new Bundle().register(HelloDag.class));
  }
}

Build the bundle into build/bundle/:

./gradlew bundle

Copy build/bundle/ into a Dag bundle, such as the default dags-folder bundle, and send the java queue to the Java coordinator in airflow.cfg:

[sdk]
coordinators = {"java": {"classpath": "airflow.sdk.coordinators.java.JavaCoordinator"}}
queue_to_coordinator = {"java": "java"}

Airflow reads the bundle like any other Dag file: java_hello shows up in the UI, runs on its schedule, and report receives the value extract returned.

Learn more

The Java SDK guide covers:

Topic Guide section
Task inputs, options, order-only dependencies, task groups, branching, triggering another Dag Declaring a Dag in Java
@Airflow.TaskHandler for the @task.stub tasks of a Python Dag, and how arguments bind Implementing stub tasks of a Python Dag
Client, Context, and what a task returns Writing tasks
Building a bundle with Gradle or Maven Building a bundle
Where to deploy a bundle, and the JavaCoordinator options Deploying
What a Dag declared in Java cannot do yet Limitations

See example/ for a working project.

Compatibility matrix

Which Airflow TaskInstance states and capabilities this SDK supports. This table is generated from capabilities.yaml; the conformance dimensions are defined in the Language SDK conformance spec. Do not edit the table by hand — edit capabilities.yaml and let the update-java-sdk-readme-matrix prek hook regenerate it.

Min. Airflow version: 3.4.0 · supervisor schema: 2026-10-30

Dimension Tier Supported Since (Airflow) Since (SDK) Notes
TaskInstance states
state: success MUST ✓ 3.4.0 1.0.0
state: failed MUST ✓ 3.4.0 1.0.0
state: up_for_retry MUST ✓ 3.4.0 1.0.0 RetryTask
state: skipped SHOULD ✓ 3.4.0 1.0.0 a TriggerDagRun task that skips an existing run
state: deferred MAY ✓ 3.4.0 1.0.0 a TriggerDagRun task that waits with deferrable
state: up_for_reschedule MAY ✗ – – runtime does not emit RescheduleTask yet
state: awaiting_input MAY ✗ – – runtime does not emit AwaitInputTask yet
state: removed MAY ✓ 3.4.0 1.0.0
Task runtime
capability: task-logging MUST ✓ 3.4.0 1.0.0 SLF4J + JPL bridged to the task log
capability: xcom-read-write MUST ✓ 3.4.0 1.0.0
capability: connection-read MUST ✓ 3.4.0 1.0.0
capability: variable-read-write MUST ✓ 3.4.0 1.0.0
capability: self-contained-bundle MUST ✓ 3.4.0 1.0.0 Airflow metadata embedded in the jar artifact
capability: retry-policy MAY ✗ – – no task-facing retry-policy API yet
capability: task-state-store MAY ✓ 3.4.0 1.0.0 Client.getTaskStateStore() get/set/delete/clear
capability: asset-state-store MAY ✗ – – no task-facing asset state-store API yet
capability: asset-event-emit MAY ✗ – – runtime does not emit asset events yet
capability: asset-event-read MAY ✗ – – no task-facing asset-event API yet
capability: object-store MAY ✗ – – no object-storage API yet
capability: listeners MAY ✗ – – no hook into the Python listener API, so a task emits no OpenLineage facets of its own
Dag definition
capability: dag-definition-authoring SHOULD ✓ 3.4.0 1.0.0
capability: dag-args MUST † ✓ 3.4.0 1.0.0
capability: task-args MUST † ✓ 3.4.0 1.0.0
capability: dag-params MUST † ✗ – –
capability: dag-callbacks MAY † ✗ – –
capability: explicit-dependencies MUST † ✓ 3.4.0 1.0.0 before / after, or @Airflow.Deps
capability: edge-labels MAY † ✓ 3.4.0 1.0.0 Label.of(text).on(task)
capability: taskflow-dependencies MUST † ✓ 3.4.0 1.0.0
capability: task-group MAY † ✓ 3.4.0 1.0.0
capability: task-loops MAY † ✗ – –
capability: if-else-branching SHOULD † ✓ 3.4.0 1.0.0 DagDef.If or @Airflow.If
capability: switch-case-branching SHOULD † ✓ 3.4.0 1.0.0 DagDef.Switch or @Airflow.Switch
capability: trigger-dag-run MAY † ✓ 3.4.0 1.0.0 TriggerDagRun
capability: dynamic-dag-generation MAY † ✓ 3.4.0 1.0.0
capability: human-in-the-loop MAY † ✗ – –
capability: dag-test SHOULD † ✗ – –
capability: asset-definition MAY † ✗ – –
capability: asset-scheduling MAY † ✗ – –
capability: mapped-tasks MAY † ✗ – –
capability: asset-inlets-outlets MAY † ✗ – –
capability: task-callbacks MAY † ✗ – – no SDK-defined task callbacks
Python Dag with stub TaskHandler
capability: mixed-lang-stub-target MUST ✓ 3.4.0 1.0.0 @task.stub
capability: taskflow-binding MUST ✓ 3.4.0 1.0.0 mapped stubs excluded
capability: mapped-tasks MAY ✗ – – Airflow does not bind arguments to mapped stubs yet
capability: asset-inlets-outlets MAY ✗ – – runtime does not report task outlets yet
capability: task-callbacks MAY ✗ – – a Python-declared task callback does not run for a stub task

Marks: ✓ supported · ✗ not supported · n/a not applicable. A tier marked † applies only when dag-definition-authoring is supported.