mirror of
https://github.com/apache/airflow.git
synced 2026-11-04 07:12:48 +00:00
Add Java SDK capability manifest and compatibility matrix (#71151)
* Add Java SDK capability manifest and compatibility matrix The Language SDK conformance spec defines which TaskInstance states and capabilities a Language SDK may declare, but nothing lets an SDK say what it actually supports. Readers of the Java SDK docs cannot tell which parts of the spec the runtime implements today — and the gaps are real and moving (native Dag authoring, deferral, and the state stores are not there yet), so an unqualified "see the conformance spec" overstates what a Java task can do. The manifest is hand-authored YAML rather than something derived from the SDK's own sources: it describes the SDK instead of being part of it, so it has no business shipping in a user's runtime dependencies, and deriving it would mean a JDK and a Gradle run in a hook that only ever renders a table. It therefore sits above every subproject in settings.gradle.kts, where no source set can pick it up. Validation carries the weight a compiler would have: unknown keys are rejected along with missing ones, since it is the only thing standing between a typo and a wrong published table. A prek hook regenerates the contributor-facing README table and the Dokka module doc from the manifest, so a capability landing in the runtime is a one-line edit rather than three tables to update by hand. The shared schema, SDK registry, and renderer let the Go and TypeScript SDKs declare theirs the same way. * Keep Language SDK compatibility matrices accurate Contradictory native-Dag declarations could pass validation while being rendered as not applicable. Published matrix tables also need to remain readable without forcing unrelated headers onto one line.
This commit is contained in:
Jason(Zhe-You) Liu authored and 
GitHub committed
1 parent
637b219934
commit
b304b4522e
11 files changed
+1023
No files matched your search
@@ -580,6 +580,58 @@ Close the vote, **drop** the staging repository in Nexus, remove the `dist/dev`
|
||||
candidate, fix the issue, and cut the next RC (`...-rc2`). The released version
|
||||
stays the same (e.g. `<VERSION>`); only the RC counter in the tag increments.
|
||||
|
||||
## Compatibility matrix
|
||||
|
||||
Which Airflow TaskInstance states and capabilities this SDK supports. This table is generated from
|
||||
[`capabilities.yaml`](capabilities.yaml); the conformance dimensions are defined in the
|
||||
[Language SDK conformance spec](https://github.com/apache/airflow/blob/main/contributing-docs/30_new_language_sdk.rst).
|
||||
Do not edit the table by hand — edit `capabilities.yaml` and let the `update-java-sdk-readme-matrix`
|
||||
prek hook regenerate it.
|
||||
|
||||
<!-- BEGIN AUTO-GENERATED LANG-SDK COMPAT MATRIX -->
|
||||
|
||||
*Min. Airflow version: 3.3 · supervisor schema: 2026-06-16*
|
||||
|
||||
| Dimension | Tier | Supported | Since | Notes |
|
||||
|---|---|---|---|---|
|
||||
| **TaskInstance states** | | | | |
|
||||
| state: `success` | MUST | ✓ | 3.3 | |
|
||||
| state: `failed` | MUST | ✓ | 3.3 | |
|
||||
| state: `up_for_retry` | MUST | ✓ | 3.3 | RetryTask |
|
||||
| state: `skipped` | SHOULD | ✗ | – | runtime does not emit TaskState skipped yet |
|
||||
| state: `deferred` | MAY | ✗ | – | runtime does not emit DeferTask yet |
|
||||
| state: `up_for_reschedule` | MAY | ✗ | – | runtime does not emit RescheduleTask yet |
|
||||
| state: `awaiting_input` | MAY | ✗ | – | runtime does not emit AwaitInputTask yet |
|
||||
| state: `removed` | MAY | ✓ | 3.3 | |
|
||||
| **Runtime capabilities** | | | | |
|
||||
| capability: `mixed-lang-stub-target` | MUST | ✓ | 3.3 | @task.stub |
|
||||
| capability: `task-logging` | MUST | ✓ | 3.3 | SLF4J + JPL bridged to the task log |
|
||||
| capability: `xcom-read-write` | MUST | ✓ | 3.3 | |
|
||||
| capability: `connection-read` | MUST | ✓ | 3.3 | |
|
||||
| capability: `variable-read-write` | MUST | ✗ | – | getVariable only; no write over the comm socket yet |
|
||||
| capability: `self-contained-bundle` | MUST | ✓ | 3.3 | Airflow metadata embedded in the jar artifact |
|
||||
| capability: `retry-policy` | MAY | ✗ | – | no task-facing retry-policy API yet |
|
||||
| capability: `task-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
|
||||
| capability: `asset-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
|
||||
| capability: `asset-event-emit` | MAY | ✗ | – | runtime does not emit asset events yet |
|
||||
| capability: `asset-event-read` | MAY | ✗ | – | no task-facing asset-event API yet |
|
||||
| **Native-Dag authoring** | | | | |
|
||||
| capability: `native-dag-authoring` | SHOULD | ✗ | – | native Dag authoring not implemented yet |
|
||||
| capability: `task-args` | MUST † | n/a | – | |
|
||||
| capability: `dag-params` | MUST † | n/a | – | |
|
||||
| capability: `taskflow-dependencies` | MUST † | n/a | – | |
|
||||
| capability: `branching` | SHOULD † | n/a | – | |
|
||||
| capability: `dag-test` | SHOULD † | n/a | – | |
|
||||
| capability: `task-group` | MAY † | n/a | – | |
|
||||
| capability: `dynamic-task-mapping` | MAY † | n/a | – | |
|
||||
| capability: `asset-inlets-outlets` | MAY † | n/a | – | |
|
||||
| capability: `asset-scheduling` | MAY † | n/a | – | |
|
||||
| capability: `object-store` | MAY † | n/a | – | |
|
||||
|
||||
*Marks: ✓ supported · ✗ not supported · n/a not applicable. A tier marked † applies only when `native-dag-authoring` is supported.*
|
||||
|
||||
<!-- END AUTO-GENERATED LANG-SDK COMPAT MATRIX -->
|
||||
|
||||
## Contributing
|
||||
|
||||
The user implements a Java application containing task methods annotated (or
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
---
|
||||
sdk: java
|
||||
|
||||
min_airflow_version: "3.3"
|
||||
|
||||
# Keep in sync with airflowSupervisorSchemaVersion in gradle.properties, which is what stamps the
|
||||
# JAR manifest. The render hook fails if the two disagree.
|
||||
supervisor_schema_version: "2026-06-16"
|
||||
|
||||
# The runtime terminates a task with SucceedTask, RetryTask, or TaskState (failed/removed); it does
|
||||
# not yet emit skipped, DeferTask, RescheduleTask, or AwaitInputTask.
|
||||
states:
|
||||
success:
|
||||
supported: true
|
||||
since: "3.3"
|
||||
failed:
|
||||
supported: true
|
||||
since: "3.3"
|
||||
up_for_retry:
|
||||
supported: true
|
||||
since: "3.3"
|
||||
note: "RetryTask"
|
||||
skipped:
|
||||
supported: false
|
||||
note: "runtime does not emit TaskState skipped yet"
|
||||
deferred:
|
||||
supported: false
|
||||
note: "runtime does not emit DeferTask yet"
|
||||
up_for_reschedule:
|
||||
supported: false
|
||||
note: "runtime does not emit RescheduleTask yet"
|
||||
awaiting_input:
|
||||
supported: false
|
||||
note: "runtime does not emit AwaitInputTask yet"
|
||||
removed:
|
||||
supported: true
|
||||
since: "3.3"
|
||||
|
||||
# Runtime capabilities reflect the task-facing Client surface; native-Dag authoring is not
|
||||
# implemented yet, so every native capability is unsupported.
|
||||
capabilities:
|
||||
mixed-lang-stub-target:
|
||||
supported: true
|
||||
since: "3.3"
|
||||
note: "@task.stub"
|
||||
task-logging:
|
||||
supported: true
|
||||
since: "3.3"
|
||||
note: "SLF4J + JPL bridged to the task log"
|
||||
xcom-read-write:
|
||||
supported: true
|
||||
since: "3.3"
|
||||
connection-read:
|
||||
supported: true
|
||||
since: "3.3"
|
||||
variable-read-write:
|
||||
supported: false
|
||||
note: "getVariable only; no write over the comm socket yet"
|
||||
self-contained-bundle:
|
||||
supported: true
|
||||
since: "3.3"
|
||||
note: "Airflow metadata embedded in the jar artifact"
|
||||
retry-policy:
|
||||
supported: false
|
||||
note: "no task-facing retry-policy API yet"
|
||||
task-state-store:
|
||||
supported: false
|
||||
note: "no task-facing state-store API yet"
|
||||
asset-state-store:
|
||||
supported: false
|
||||
note: "no task-facing state-store API yet"
|
||||
asset-event-emit:
|
||||
supported: false
|
||||
note: "runtime does not emit asset events yet"
|
||||
asset-event-read:
|
||||
supported: false
|
||||
note: "no task-facing asset-event API yet"
|
||||
native-dag-authoring:
|
||||
supported: false
|
||||
note: "native Dag authoring not implemented yet"
|
||||
task-args:
|
||||
supported: false
|
||||
dag-params:
|
||||
supported: false
|
||||
taskflow-dependencies:
|
||||
supported: false
|
||||
branching:
|
||||
supported: false
|
||||
dag-test:
|
||||
supported: false
|
||||
task-group:
|
||||
supported: false
|
||||
dynamic-task-mapping:
|
||||
supported: false
|
||||
asset-inlets-outlets:
|
||||
supported: false
|
||||
asset-scheduling:
|
||||
supported: false
|
||||
object-store:
|
||||
supported: false
|
||||
@@ -267,7 +267,15 @@ sourceSets {
|
||||
|
||||
dokka {
|
||||
moduleVersion.set(project.version.toString())
|
||||
pluginsConfiguration.html {
|
||||
// Widens the narrow compatibility-matrix columns; see the comments in the file.
|
||||
customStyleSheets.from(layout.projectDirectory.file("dokka/matrix.css"))
|
||||
}
|
||||
dokkaSourceSets.configureEach {
|
||||
// Module-level documentation, including the generated Language SDK compatibility matrix.
|
||||
// Dokka rejects the file unless "# Module sdk" is its very first line, so module.md carries
|
||||
// the ASF license header just below the heading instead of above it.
|
||||
includes.from("module.md")
|
||||
// Suppress everything in 'execution' since it's implementation detail.
|
||||
perPackageOption {
|
||||
matchingRegex = """org\.apache\.airflow\.sdk\.execution.*"""
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
/*!
|
||||
* Licensed to the Apache Software Foundation (ASF) under one
|
||||
* or more contributor license agreements. See the NOTICE file
|
||||
* distributed with this work for additional information
|
||||
* regarding copyright ownership. The ASF licenses this file
|
||||
* to you under the Apache License, Version 2.0 (the
|
||||
* "License"); you may not use this file except in compliance
|
||||
* with the License. You may obtain a copy of the License at
|
||||
*
|
||||
* http://www.apache.org/licenses/LICENSE-2.0
|
||||
*
|
||||
* Unless required by applicable law or agreed to in writing,
|
||||
* software distributed under the License is distributed on an
|
||||
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
* KIND, either express or implied. See the License for the
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
|
||||
/* Tier ("SHOULD †"), Supported ("n/a") and Since ("3.3") in the compatibility matrix. */
|
||||
.table--container th:nth-child(2),
|
||||
.table--container th:nth-child(3),
|
||||
.table--container td:nth-child(2),
|
||||
.table--container td:nth-child(3) {
|
||||
min-width: 9ch;
|
||||
}
|
||||
|
||||
.table--container th:nth-child(4),
|
||||
.table--container td:nth-child(4) {
|
||||
min-width: 6ch;
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
# Module sdk
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
The Apache Airflow Java SDK — author and run Airflow task implementations in JVM languages.
|
||||
|
||||
## Language SDK compatibility matrix
|
||||
|
||||
Which Airflow TaskInstance states and capabilities the Java SDK currently supports. The normative
|
||||
meaning of each dimension is defined in the
|
||||
[Language SDK conformance specification](https://github.com/apache/airflow/blob/main/contributing-docs/30_new_language_sdk.rst).
|
||||
|
||||
<!-- BEGIN AUTO-GENERATED LANG-SDK COMPAT MATRIX -->
|
||||
|
||||
*Min. Airflow version: 3.3 · supervisor schema: 2026-06-16*
|
||||
|
||||
| Dimension | Tier | Supported | Since | Notes |
|
||||
|---|---|---|---|---|
|
||||
| **TaskInstance states** | | | | |
|
||||
| state: `success` | MUST | ✓ | 3.3 | |
|
||||
| state: `failed` | MUST | ✓ | 3.3 | |
|
||||
| state: `up_for_retry` | MUST | ✓ | 3.3 | RetryTask |
|
||||
| state: `skipped` | SHOULD | ✗ | – | runtime does not emit TaskState skipped yet |
|
||||
| state: `deferred` | MAY | ✗ | – | runtime does not emit DeferTask yet |
|
||||
| state: `up_for_reschedule` | MAY | ✗ | – | runtime does not emit RescheduleTask yet |
|
||||
| state: `awaiting_input` | MAY | ✗ | – | runtime does not emit AwaitInputTask yet |
|
||||
| state: `removed` | MAY | ✓ | 3.3 | |
|
||||
| **Runtime capabilities** | | | | |
|
||||
| capability: `mixed-lang-stub-target` | MUST | ✓ | 3.3 | @task.stub |
|
||||
| capability: `task-logging` | MUST | ✓ | 3.3 | SLF4J + JPL bridged to the task log |
|
||||
| capability: `xcom-read-write` | MUST | ✓ | 3.3 | |
|
||||
| capability: `connection-read` | MUST | ✓ | 3.3 | |
|
||||
| capability: `variable-read-write` | MUST | ✗ | – | getVariable only; no write over the comm socket yet |
|
||||
| capability: `self-contained-bundle` | MUST | ✓ | 3.3 | Airflow metadata embedded in the jar artifact |
|
||||
| capability: `retry-policy` | MAY | ✗ | – | no task-facing retry-policy API yet |
|
||||
| capability: `task-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
|
||||
| capability: `asset-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
|
||||
| capability: `asset-event-emit` | MAY | ✗ | – | runtime does not emit asset events yet |
|
||||
| capability: `asset-event-read` | MAY | ✗ | – | no task-facing asset-event API yet |
|
||||
| **Native-Dag authoring** | | | | |
|
||||
| capability: `native-dag-authoring` | SHOULD | ✗ | – | native Dag authoring not implemented yet |
|
||||
| capability: `task-args` | MUST † | n/a | – | |
|
||||
| capability: `dag-params` | MUST † | n/a | – | |
|
||||
| capability: `taskflow-dependencies` | MUST † | n/a | – | |
|
||||
| capability: `branching` | SHOULD † | n/a | – | |
|
||||
| capability: `dag-test` | SHOULD † | n/a | – | |
|
||||
| capability: `task-group` | MAY † | n/a | – | |
|
||||
| capability: `dynamic-task-mapping` | MAY † | n/a | – | |
|
||||
| capability: `asset-inlets-outlets` | MAY † | n/a | – | |
|
||||
| capability: `asset-scheduling` | MAY † | n/a | – | |
|
||||
| capability: `object-store` | MAY † | n/a | – | |
|
||||
|
||||
*Marks: ✓ supported · ✗ not supported · n/a not applicable. A tier marked † applies only when `native-dag-authoring` is supported.*
|
||||
|
||||
<!-- END AUTO-GENERATED LANG-SDK COMPAT MATRIX -->
|
||||
Reference in new issue
Block a user