mirror of
https://github.com/apache/airflow.git
synced 2026-11-04 16:01:19 +00:00
eda782f76eaeebcd41f47869ec249a58c6db8fb4
6
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  PoAn YangandZHE YOU LIU
|
5a64dfcc78 |
Java SDK: Add edge labels to native Dags (#74523)
* Java SDK: Add edge labels to native Dags Signed-off-by: PoAn Yang <payang@apache.org> * Apply edge labels in the order they were set A task edge's own label always won over a group edge's label, and a relabeled group edge kept its first position, so edge_info differed from Python's, where the label set last wins. Also note that Python's Label rewrites an edge when one end is in no task group. * Java SDK: Write the edge label text before the task Signed-off-by: PoAn Yang <payang@apache.org> --------- Signed-off-by: PoAn Yang <payang@apache.org> Co-authored-by: ZHE YOU LIU <zhu424.dev@gmail.com> |
||
 Jason(Zhe-You) Liu
|
c5f504366d |
Java SDK: Trigger another Dag's run from a native Java Dag (#74440)
* Java SDK: Trigger another Dag's run from a native Java Dag
A native Java Dag could not start another Dag's run. Airflow's
TriggerDagRunOperator is Python, and a Java bundle has no Python Dag file to
declare one in, so a Java Dag that fans out to another Dag needed a Python
counterpart for that one task.
A `TriggerDagRun` passed to `dag.task` declares a task that has no Java body
to run; the Java runtime runs it as `TriggerDagRunOperator` does. Its
settings are named as Python names the operator's parameters:
```java
dag.task("trigger_downstream",
new TriggerDagRun("downstream_etl")
.config("wait_for_completion", true)
.config("conf", Map.of("rows", 2)));
```
An annotated Dag declares one from an ordinary `@Builder.Task` method that
returns a `TriggerDagRun`. The serialized Dag has to carry what is
triggered, so that method runs when the Dag is built rather than when the
task runs, and therefore takes no parameters.
The task serializes as Python's own `TriggerDagRunOperator`: its template
fields, its UI colour and extra link, and a `dag_dependencies` entry, so the
Airflow UI draws the dependency between the two Dags. Python writes only an
operator's template fields and keeps the rest in the Dag file; a Java Dag
has no such file, so every setting the author made is written too.
The runtime records the triggered run's link and ID as XComs, skips or fails
on a run that already exists, and waits for completion either by polling or,
with `deferrable`, by deferring to the Python triggerer's DagStateTrigger and
resuming from the event it fires. That makes this the first Java runtime to
emit DeferTask and a skipped TaskState.
`scripts/ci/lang_sdk_serialization/test_dags.yaml` now gives the shared
trigger Dag a real logical date instead of a Jinja template, because the SDK
takes one as a date-time. The Go SDK is making the same change in #74417;
whichever lands second drops that one line, and until then the Go
serialization conformance check reports that single difference.
* Java SDK: Point both example triggers at a Dag that triggers nothing
* Java SDK: Say why a switch cannot choose a task that triggers a Dag run
* Java SDK: Snapshot a TriggerDagRun when its task is declared
* Java SDK: Check a TriggerDagRun setting the way every other config key is checked
* Java SDK: Describe what a trigger task pushes and how it names a run
* Java SDK: Correct which runtimes already run trigger tasks
* Java SDK: Format a trigger task's date-times in one place
* Java SDK: Reject a trigger method that throws a checked exception
* Java SDK: Give the trigger fixture a logical date every SDK can write
* Java SDK: Run a trigger task through runTask in tests
* Java SDK: Check a trigger task polls the run it started
|
||
|
Jason(Zhe-You) Liu
|
c3043c266c |
Java SDK: Choose one of several tasks with a branch (#74439)
* Java SDK: Choose one of several tasks with a branch A condition picks between two tasks, but a Dag that routes work by size or by region needs more than two, and splitting that into nested conditions makes the graph unreadable. `DagDef.Branch` and `@Builder.Branch` declare a task that chooses one of the tasks listed with `option(...)`; every other one is skipped. The chosen task's ID becomes the branch's return value, as it does for Python's branch operators. A branch chooses exactly one case, so a task that runs after several of them needs a trigger rule such as `none_failed_min_one_success`. Each authoring style names the case in the way that style can type-check. A `BranchTask` returns the case's own class, so javac checks it; two cases that share a class are rejected where the second is named, since the branch could not tell them apart. A `@Builder.Branch` method has no such class to name -- a task method's implementation is generated -- so it returns a `TaskId` from the generated `<Dag>Builder.TaskIds`, which holds one constant per task of the Dag. Either way nothing hard-codes a task id string, and the generated holder is emitted only for a Dag that declares a branch. * Java SDK: Name the multi-way branch Switch, as the other SDKs do * Java SDK: Name a switch case by its task class in both styles * Java SDK: Say when a switch rejects a choice and when it fails * Java SDK: Cover a generated switch that names a task it cannot choose * Java SDK: Check a switch carries the chosen task's id * Java SDK: Let a wiring view name a switch's cases in more than one call |
||
|
Jason(Zhe-You) Liu
|
d4013cc2b5 |
Java SDK: Pick one of two tasks with a condition (#74438)
* Java SDK: Pick one of two tasks with a condition A native Java Dag could only run every task it declared: it had no way to let a task's result decide what runs next, so a Dag that loads only when there are rows had to be split in two or fall back to a Python stub. `DagDef.If` and `@Builder.If` declare a task whose boolean names the side that runs; the other is skipped. The deciding task carries `_can_skip_downstream`, so Airflow skips a cleared side again rather than running it, and the SDK writes the `skipmixin_key` XCom before the skip, as Python's SkipMixin does. It skips only the side not taken, so a task after both sides needs a trigger rule such as `none_failed_min_one_success`. `If` is capitalized because `if` is a Java keyword and `@Builder.If` reads the same way. With no task id given, one is derived from the condition class's simple name, so `HasRows.class` becomes `hasRows`. The runtime reaches `SkipDownstreamTasks` through the transport client only, never through the task-facing `Client`, so a task body cannot skip tasks the Dag did not declare it may. * Java SDK: Let a wiring view name a condition's sides in more than one call * Java SDK: Reject a decider class registered as an ordinary task * Java SDK: Name a condition's sides Then and Else * Java SDK: Pin the order a condition skips and records what it skipped * Java SDK: Drop the unused switch in the condition test's decider |
||
|
Jason(Zhe-You) Liu
|
01b977568e |
Java SDK: Serialize native Dags to DagSerialization v3 (#71190)
* Java SDK: Serialize native Dags to DagSerialization v3 on parse requests A Java-authored Dag could not reach the scheduler on its own: the runtime answered task-execution requests only, so a Python stub file still had to exist purely to describe the Dag's structure. With dependency edges and schema-keyed configuration now recorded on the Java Dag model, the runtime has everything it needs to answer the coordinator's DagFileParseRequest the same way the Go SDK does, and the nativedag examples become real schedulable Dags with no Python counterpart. Native Java tasks deliberately emit no _arg_bindings: the execution API delivers bindings only for Python _StubOperator tasks, and a Java task always runs inside the JVM bundle that already holds its wired inputs, so the runtime resolves them locally. Cron schedules map to CronTriggerTimetable only, mirroring the Go SDK until the supervisor forwards the [scheduler] timetable flags over the coordinator protocol (see the TODO at the timetable serializer). * Test that the conformance check compares Dag params * Java SDK: Leave config-backed Dag fields for Airflow to fill in * Java SDK: Leave out the email flags a Java task cannot act on * Java SDK: Serialize a native Dag's task groups * Java SDK: Write each Dag tag once, as Python does * Java SDK: Check native Dag serialization against Airflow's * Java SDK: Run the serialization conformance check on every file that steers it `Bundle.register` expands the group edges, `Refs` decides which group each task lands in, and `sdk/build.gradle.kts` generates the schema fields the serializer writes from. All three change the serialized output, so add them to the hook's trigger set. Report a failed classpath build with the Gradle output instead of an `IndexError` on an empty line list or a bare `CalledProcessError`. * Java SDK: Build a native Dag's timetable the way Python does Expand a cron preset before writing it, so `@daily` serializes as `0 0 * * *` and the Dag hashes the same as the Python one. Reject a schedule that is neither a preset nor a cron expression where the Dag is written, rather than letting it reach a scheduler that cannot build a timetable from it. Take the timezone from the Dag's start date, as Python's `DAG.timezone` does, instead of always writing UTC: a Dag started at a non-UTC offset now fires at that offset rather than at the same wall clock in UTC. * Java SDK: Carry a native Dag's call arguments as argument bindings A native Dag's serialized form recorded its task edges but nothing about what each task was called with, so changing a literal argument left the Dag byte-identical: no new version, and nothing in the version diff. Write the same `is_stub` flag and `_arg_bindings` spec the TypeScript SDK writes, as ADR-0007 decision G says a natively authored Dag should. The wiring view now passes its parameter names along with the arguments, so each binding can name the parameter it feeds. A Java task still resolves its arguments from the Dag in its own bundle rather than reading the spec back: the bundle that runs the task also holds the call that wired it, so the values never have to travel, and a `TaskInput` keeps binding as one whole input. The spec is what Airflow records and shows. Since the spec travels as JSON, a literal with no JSON form is now rejected where the Dag is parsed instead of reaching a task that cannot receive it. * Java SDK: Keep one unserializable Dag from failing a whole bundle's parse A Dag that could not be serialized threw out of the parse response, so every other Dag in the same JAR vanished with it and `import_errors` was never filled. Report the failure under the bundle-relative path the way the TypeScript SDK does, naming the Dag, and serialize the rest. * Java SDK: Mark the native Dag capabilities as shipping in 3.4 `native-dag-authoring`, `task-args`, `taskflow-dependencies` and `task-group` are new in this release, not in 3.3, which is what the matrix's "supported since" column says. * Java SDK: Accept every cron schedule croniter does and bind wired inputs first The cron shape check now accepts comma lists of names, month and weekday names, the `#`, `L` and `W` qualifiers and the `@midnight` and `@annually` aliases, which Python stores unexpanded. Each of these previously turned the Dag into an import error. The schema-default comment records why explicit default values are dropped, and the timezone TODO and the Java docs note that a cron Dag with no start date is scheduled in UTC. A wired input wins over a runtime binding in ArgValues. The Java docs and ADR 0007 now say so, and a test pins it. The conformance run now passes `--supports literal_inputs` and the Java serializer wires each task's upstream handles and literals as call arguments, so the run exercises `_arg_bindings`. The hook's file filter covers `internal/Fields.kt`. Selective checks skip the hook unless a java-sdk file, Airflow's serializer or schema, or the shared harness changed, through a new file group because serializer changes do not force full tests. |
||
 Tzu-ping Chung
|
b0300009a4 | Verify Java SDK artifact integrity continuously (#69643) |