mirror of
https://github.com/apache/airflow.git
synced 2026-11-04 07:12:48 +00:00
Jason(Zhe-You) Liu
01b977568e
* Java SDK: Serialize native Dags to DagSerialization v3 on parse requests A Java-authored Dag could not reach the scheduler on its own: the runtime answered task-execution requests only, so a Python stub file still had to exist purely to describe the Dag's structure. With dependency edges and schema-keyed configuration now recorded on the Java Dag model, the runtime has everything it needs to answer the coordinator's DagFileParseRequest the same way the Go SDK does, and the nativedag examples become real schedulable Dags with no Python counterpart. Native Java tasks deliberately emit no _arg_bindings: the execution API delivers bindings only for Python _StubOperator tasks, and a Java task always runs inside the JVM bundle that already holds its wired inputs, so the runtime resolves them locally. Cron schedules map to CronTriggerTimetable only, mirroring the Go SDK until the supervisor forwards the [scheduler] timetable flags over the coordinator protocol (see the TODO at the timetable serializer). * Test that the conformance check compares Dag params * Java SDK: Leave config-backed Dag fields for Airflow to fill in * Java SDK: Leave out the email flags a Java task cannot act on * Java SDK: Serialize a native Dag's task groups * Java SDK: Write each Dag tag once, as Python does * Java SDK: Check native Dag serialization against Airflow's * Java SDK: Run the serialization conformance check on every file that steers it `Bundle.register` expands the group edges, `Refs` decides which group each task lands in, and `sdk/build.gradle.kts` generates the schema fields the serializer writes from. All three change the serialized output, so add them to the hook's trigger set. Report a failed classpath build with the Gradle output instead of an `IndexError` on an empty line list or a bare `CalledProcessError`. * Java SDK: Build a native Dag's timetable the way Python does Expand a cron preset before writing it, so `@daily` serializes as `0 0 * * *` and the Dag hashes the same as the Python one. Reject a schedule that is neither a preset nor a cron expression where the Dag is written, rather than letting it reach a scheduler that cannot build a timetable from it. Take the timezone from the Dag's start date, as Python's `DAG.timezone` does, instead of always writing UTC: a Dag started at a non-UTC offset now fires at that offset rather than at the same wall clock in UTC. * Java SDK: Carry a native Dag's call arguments as argument bindings A native Dag's serialized form recorded its task edges but nothing about what each task was called with, so changing a literal argument left the Dag byte-identical: no new version, and nothing in the version diff. Write the same `is_stub` flag and `_arg_bindings` spec the TypeScript SDK writes, as ADR-0007 decision G says a natively authored Dag should. The wiring view now passes its parameter names along with the arguments, so each binding can name the parameter it feeds. A Java task still resolves its arguments from the Dag in its own bundle rather than reading the spec back: the bundle that runs the task also holds the call that wired it, so the values never have to travel, and a `TaskInput` keeps binding as one whole input. The spec is what Airflow records and shows. Since the spec travels as JSON, a literal with no JSON form is now rejected where the Dag is parsed instead of reaching a task that cannot receive it. * Java SDK: Keep one unserializable Dag from failing a whole bundle's parse A Dag that could not be serialized threw out of the parse response, so every other Dag in the same JAR vanished with it and `import_errors` was never filled. Report the failure under the bundle-relative path the way the TypeScript SDK does, naming the Dag, and serialize the rest. * Java SDK: Mark the native Dag capabilities as shipping in 3.4 `native-dag-authoring`, `task-args`, `taskflow-dependencies` and `task-group` are new in this release, not in 3.3, which is what the matrix's "supported since" column says. * Java SDK: Accept every cron schedule croniter does and bind wired inputs first The cron shape check now accepts comma lists of names, month and weekday names, the `#`, `L` and `W` qualifiers and the `@midnight` and `@annually` aliases, which Python stores unexpanded. Each of these previously turned the Dag into an import error. The schema-default comment records why explicit default values are dropped, and the timezone TODO and the Java docs note that a cron Dag with no start date is scheduled in UTC. A wired input wins over a runtime binding in ArgValues. The Java docs and ADR 0007 now say so, and a test pins it. The conformance run now passes `--supports literal_inputs` and the Java serializer wires each task's upstream handles and literals as call arguments, so the run exercises `_arg_bindings`. The hook's file filter covers `internal/Fields.kt`. Selective checks skip the hook unless a java-sdk file, Airflow's serializer or schema, or the shared harness changed, through a new file group because serializer changes do not force full tests.
124 lines
3.4 KiB
YAML
124 lines
3.4 KiB
YAML
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing,
|
|
# software distributed under the License is distributed on an
|
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
# KIND, either express or implied. See the License for the
|
|
# specific language governing permissions and limitations
|
|
# under the License.
|
|
---
|
|
sdk: java
|
|
|
|
min_airflow_version: "3.3"
|
|
|
|
# Keep in sync with airflowSupervisorSchemaVersion in gradle.properties, which is what stamps the
|
|
# JAR manifest. The render hook fails if the two disagree.
|
|
supervisor_schema_version: "2026-10-30"
|
|
|
|
# The runtime terminates a task with SucceedTask, RetryTask, or TaskState (failed/removed); it does
|
|
# not yet emit skipped, DeferTask, RescheduleTask, or AwaitInputTask.
|
|
states:
|
|
success:
|
|
supported: true
|
|
since: "3.3"
|
|
failed:
|
|
supported: true
|
|
since: "3.3"
|
|
up_for_retry:
|
|
supported: true
|
|
since: "3.3"
|
|
note: "RetryTask"
|
|
skipped:
|
|
supported: false
|
|
note: "runtime does not emit TaskState skipped yet"
|
|
deferred:
|
|
supported: false
|
|
note: "runtime does not emit DeferTask yet"
|
|
up_for_reschedule:
|
|
supported: false
|
|
note: "runtime does not emit RescheduleTask yet"
|
|
awaiting_input:
|
|
supported: false
|
|
note: "runtime does not emit AwaitInputTask yet"
|
|
removed:
|
|
supported: true
|
|
since: "3.3"
|
|
|
|
# Runtime capabilities reflect the task-facing Client surface. Native-Dag capabilities describe
|
|
# what a Java-authored Dag can declare and deliver through Dag serialization.
|
|
capabilities:
|
|
mixed-lang-stub-target:
|
|
supported: true
|
|
since: "3.3"
|
|
note: "@task.stub"
|
|
taskflow-binding:
|
|
supported: false
|
|
note: "bind @task.stub literal/XCom args to the native handler"
|
|
task-logging:
|
|
supported: true
|
|
since: "3.3"
|
|
note: "SLF4J + JPL bridged to the task log"
|
|
xcom-read-write:
|
|
supported: true
|
|
since: "3.3"
|
|
connection-read:
|
|
supported: true
|
|
since: "3.3"
|
|
variable-read-write:
|
|
supported: true
|
|
since: "3.3"
|
|
self-contained-bundle:
|
|
supported: true
|
|
since: "3.3"
|
|
note: "Airflow metadata embedded in the jar artifact"
|
|
retry-policy:
|
|
supported: false
|
|
note: "no task-facing retry-policy API yet"
|
|
task-state-store:
|
|
supported: true
|
|
since: "3.3"
|
|
note: "Client.getTaskStateStore() get/set/delete/clear"
|
|
asset-state-store:
|
|
supported: false
|
|
note: "no task-facing state-store API yet"
|
|
asset-event-emit:
|
|
supported: false
|
|
note: "runtime does not emit asset events yet"
|
|
asset-event-read:
|
|
supported: false
|
|
note: "no task-facing asset-event API yet"
|
|
native-dag-authoring:
|
|
supported: true
|
|
since: "3.4"
|
|
task-args:
|
|
supported: true
|
|
since: "3.4"
|
|
dag-params:
|
|
supported: false
|
|
taskflow-dependencies:
|
|
supported: true
|
|
since: "3.4"
|
|
branching:
|
|
supported: false
|
|
dag-test:
|
|
supported: false
|
|
task-group:
|
|
supported: true
|
|
since: "3.4"
|
|
dynamic-task-mapping:
|
|
supported: false
|
|
asset-inlets-outlets:
|
|
supported: false
|
|
asset-scheduling:
|
|
supported: false
|
|
object-store:
|
|
supported: false
|