* Fix Dag id collision in the Java SDK example bundle
The interface example registered its task handlers as a Java Dag, so the
example jar declared a Dag with the same id as the Python Dag that owns
those tasks. When the jar is also a Dag bundle, the jar's Dag replaced
the Python one and its runs failed. The new java_task_handler_ prefix
also tells these Dags apart from the Java-defined java_native_ ones.
Co-Authored-By: Claude <noreply@anthropic.com>
* Register the interface example handlers inline
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Java SDK: Rewrite the user docs around Dag definition
Reorder the Java SDK guide to start from a Dag defined in Java, document
conditions, switches and TriggerDagRun, and add a table of how stub task
arguments bind. Split the README into a user landing page and
DEVELOPMENT.md, and add an example README.
* Add Logback and appender to the docs spelling wordlist
* Address review: Dag-level queue, version sync, dropped constraints
Set the queue once on the Dag in every example, now that a Java Dag takes
one, and bump the documented version to 1.0.0-beta2 everywhere. Add a prek
hook that fails when the guide, the README and the Dokka landing page name
different versions. Restore the TaskInput constraints, correct the
LocalExecutor wording, and drop the Python Label comparison.
* Generate the quick start of the Java SDK README and Dokka page
The quick start appears on three pages. The guide owns it now, between
SHARED SNIPPET markers, and the sync-java-sdk-docs hook copies it into the
other two and still checks that all three install the same version.
* Address review: install a version placeholder, not a release
The three pages that show how to install the SDK pinned a release that is
not out yet, which also held the PR back. They now install `<version>` and
link to Maven Central, and the sync hook fails when a page names a release
instead. The parse-failure bullet leads with the SDK release a JAR was built
with, and keeps the supervisor schema version as the explanation.
---------
* Java SDK: Let a native Java Dag set one queue for all its tasks
Every task of a native Java Dag runs on the Java coordinator, so each one
needs a queue that queue_to_coordinator routes there. Airflow's Dag has no
queue setting, so authors had to repeat the same queue on every task. The
Go and TypeScript SDKs already take a Dag-level queue that each task
inherits unless it sets its own; this gives Java the same.
* Reject a blank Dag-level queue and fix the native Dag queue docs
A blank queue matched no coordinator, so every task stayed queued.
The parsing docs still said there was no Dag-level queue and routed a
queue the examples do not use.
* Reject a blank Dag-level queue in the annotation processor
* Carry the blank rule on the schema field, for Dag and task queues alike
* Set the queue once per Dag in the e2e native bundle
* Put the blank check on Field and tidy the comments
---------
* Java SDK: Add edge labels to native Dags
Signed-off-by: PoAn Yang <payang@apache.org>
* Apply edge labels in the order they were set
A task edge's own label always won over a group edge's label, and a
relabeled group edge kept its first position, so edge_info differed from
Python's, where the label set last wins. Also note that Python's Label
rewrites an edge when one end is in no task group.
* Java SDK: Write the edge label text before the task
Signed-off-by: PoAn Yang <payang@apache.org>
---------
Signed-off-by: PoAn Yang <payang@apache.org>
Co-authored-by: ZHE YOU LIU <zhu424.dev@gmail.com>
* Java SDK: Trigger another Dag's run from a native Java Dag
A native Java Dag could not start another Dag's run. Airflow's
TriggerDagRunOperator is Python, and a Java bundle has no Python Dag file to
declare one in, so a Java Dag that fans out to another Dag needed a Python
counterpart for that one task.
A `TriggerDagRun` passed to `dag.task` declares a task that has no Java body
to run; the Java runtime runs it as `TriggerDagRunOperator` does. Its
settings are named as Python names the operator's parameters:
```java
dag.task("trigger_downstream",
new TriggerDagRun("downstream_etl")
.config("wait_for_completion", true)
.config("conf", Map.of("rows", 2)));
```
An annotated Dag declares one from an ordinary `@Builder.Task` method that
returns a `TriggerDagRun`. The serialized Dag has to carry what is
triggered, so that method runs when the Dag is built rather than when the
task runs, and therefore takes no parameters.
The task serializes as Python's own `TriggerDagRunOperator`: its template
fields, its UI colour and extra link, and a `dag_dependencies` entry, so the
Airflow UI draws the dependency between the two Dags. Python writes only an
operator's template fields and keeps the rest in the Dag file; a Java Dag
has no such file, so every setting the author made is written too.
The runtime records the triggered run's link and ID as XComs, skips or fails
on a run that already exists, and waits for completion either by polling or,
with `deferrable`, by deferring to the Python triggerer's DagStateTrigger and
resuming from the event it fires. That makes this the first Java runtime to
emit DeferTask and a skipped TaskState.
`scripts/ci/lang_sdk_serialization/test_dags.yaml` now gives the shared
trigger Dag a real logical date instead of a Jinja template, because the SDK
takes one as a date-time. The Go SDK is making the same change in #74417;
whichever lands second drops that one line, and until then the Go
serialization conformance check reports that single difference.
* Java SDK: Point both example triggers at a Dag that triggers nothing
* Java SDK: Say why a switch cannot choose a task that triggers a Dag run
* Java SDK: Snapshot a TriggerDagRun when its task is declared
* Java SDK: Check a TriggerDagRun setting the way every other config key is checked
* Java SDK: Describe what a trigger task pushes and how it names a run
* Java SDK: Correct which runtimes already run trigger tasks
* Java SDK: Format a trigger task's date-times in one place
* Java SDK: Reject a trigger method that throws a checked exception
* Java SDK: Give the trigger fixture a logical date every SDK can write
* Java SDK: Run a trigger task through runTask in tests
* Java SDK: Check a trigger task polls the run it started
* Java SDK: Choose one of several tasks with a branch
A condition picks between two tasks, but a Dag that routes work by size or
by region needs more than two, and splitting that into nested conditions
makes the graph unreadable.
`DagDef.Branch` and `@Builder.Branch` declare a task that chooses one of
the tasks listed with `option(...)`; every other one is skipped. The chosen
task's ID becomes the branch's return value, as it does for Python's branch
operators. A branch chooses exactly one case, so a task that runs after
several of them needs a trigger rule such as
`none_failed_min_one_success`.
Each authoring style names the case in the way that style can type-check. A
`BranchTask` returns the case's own class, so javac checks it; two cases
that share a class are rejected where the second is named, since the branch
could not tell them apart. A `@Builder.Branch` method has no such class to
name -- a task method's implementation is generated -- so it returns a
`TaskId` from the generated `<Dag>Builder.TaskIds`, which holds one
constant per task of the Dag. Either way nothing hard-codes a task id
string, and the generated holder is emitted only for a Dag that declares a
branch.
* Java SDK: Name the multi-way branch Switch, as the other SDKs do
* Java SDK: Name a switch case by its task class in both styles
* Java SDK: Say when a switch rejects a choice and when it fails
* Java SDK: Cover a generated switch that names a task it cannot choose
* Java SDK: Check a switch carries the chosen task's id
* Java SDK: Let a wiring view name a switch's cases in more than one call
* Java SDK: Pick one of two tasks with a condition
A native Java Dag could only run every task it declared: it had no way to
let a task's result decide what runs next, so a Dag that loads only when
there are rows had to be split in two or fall back to a Python stub.
`DagDef.If` and `@Builder.If` declare a task whose boolean names the side
that runs; the other is skipped. The deciding task carries
`_can_skip_downstream`, so Airflow skips a cleared side again rather than
running it, and the SDK writes the `skipmixin_key` XCom before the skip, as
Python's SkipMixin does. It skips only the side not taken, so a task after
both sides needs a trigger rule such as `none_failed_min_one_success`.
`If` is capitalized because `if` is a Java keyword and `@Builder.If` reads
the same way. With no task id given, one is derived from the condition
class's simple name, so `HasRows.class` becomes `hasRows`.
The runtime reaches `SkipDownstreamTasks` through the transport client
only, never through the task-facing `Client`, so a task body cannot skip
tasks the Dag did not declare it may.
* Java SDK: Let a wiring view name a condition's sides in more than one call
* Java SDK: Reject a decider class registered as an ordinary task
* Java SDK: Name a condition's sides Then and Else
* Java SDK: Pin the order a condition skips and records what it skipped
* Java SDK: Drop the unused switch in the condition test's decider
* Java SDK: Group a native Dag's tasks with task groups
* Java SDK: Reserve the whole group view surface and pin group edge order
The reserved names of a group's wiring view are now read from Deps.TaskGroup, so
endpoints, before and after are rejected as task method names inside a group, not
only groupId and nodes. A task named before would otherwise win overload
resolution over the inherited before(Flow...) and silently register a task.
Add tests for that clash, for duplicate group IDs in one scope, and for the order
group edges resolve in: an empty group prefers an earlier edge over its parent,
and edges resolve in the order they were drawn.
Correct the Task groups docs and TaskGroupRef KDoc: only what a group holds is
read once, while edges still resolve in drawing order. Restore the missing
transform, load and audit calls in the group wiring example so it parses.
* Java SDK: Declare a Dag's task graph with @Builder.Deps
An annotated Dag class could name its tasks but not say how they connect,
so a Dag whose annotations carried everything else still needed a Python
Dag file for its graph. Declaring the graph by calling the tasks lets
javac check that each task is fed something its parameter accepts.
Bundle.register(Class) now also takes the @Builder.Dag class itself and
registers the Dag its generated builder builds, so there is still only one
name to keep in sync.
* Java SDK: Quote wired task ids as Java string literals
* Java SDK: Validate the @Builder.Deps class at compile time
* Java SDK: Reject task method names that clash with the wiring view
* Java SDK: Type a wired numeric input by its parameter's own type
* Java SDK: Register both the Dag and handlers of a class, unwrapping builder failures
* Java SDK: Reject a raw null wiring argument and test the recorder's guards
* Java SDK: Reword the native-Dag docs without em-dashes
* Java SDK: Drop Bundle tests that repeat existing coverage
* Java SDK: Require a @Builder.Deps class on every @Builder.Dag class
* Java SDK: Back the k8s test's stub tasks with task handlers
* Java SDK: Reject overloaded task and task-handler methods
* Java SDK: Fix the Dag examples a required wiring class invalidates
* Java SDK: Describe native Java Dags without reference to Python
A Dag authored entirely in Java has no Python in the picture, so teaching
its edge verbs as "Java's spelling of >> and <<" asks the reader to know a
language the feature does not involve. The mixed-language surface keeps its
Python references, because there the Python Dag file is part of what the
reader has to understand.
* Java SDK: Generate Dag and task configuration from the serialization schema
A Java-authored Dag could carry no schedule, retries, tags or any other
Airflow setting, so it could be defined but not configured. Hand-listing
those settings in Java would drift from the Python semantics they mirror,
so they are recorded and validated against Airflow's own Dag serialization
schema instead, and take effect once the runtime answers parse requests.
The `Builder` annotations are generated from a copy of that schema vendored
at java-sdk/sdk/schema/dag-schema.json, so a source release builds without
the monorepo; a prek hook keeps the copy in sync.
* Java SDK: Draw a Java-authored Dag's edges with before and after
A Dag authored in Java had no way to say what runs after what. Without a
Python Dag file to hold the graph, every task was a root, so a Dag that
Java owns end to end could not be expressed at all.
The verbs are before and after, matching the TypeScript SDK, and leaving
then free for the conditional control flow that reads as a then-branch.
They are the only way to draw an edge, so one graph has one spelling
however it was built.
* Add notes on delibrate implementation choices
---------
Co-authored-by: Tzu-ping Chung <tp@astronomer.io>
* Java SDK: Register tasks as first-class TaskDef objects
`dag.addTask("extract", Extract.class)` stored tasks as a plain
`Map<String, Class<out Task>>`, which leaves nowhere to hang anything
else a task needs: dependency edges, task-level configuration, and
argument wiring all have to attach to a per-task object, and a map of
classes cannot carry them. Introducing that object now keeps those
follow-ups additive instead of forcing another break of the registration
API later.
The annotation surface keeps `Builder.Dag` / `Builder.Task`, and the
interface users implement keeps the `Task` name, so the definition
objects are `DagDef` and `TaskDef` -- a pairing that stays unambiguous
next to `Task` at a use site. The SDK is pre-1.0, so the old
string-keyed overload is removed outright rather than deprecated.
The KubernetesExecutor lang-SDK example adopts the new names directly rather
than pinning to the old ones: CI now builds those artifacts from the branch
under test (#71527), so the example no longer has to compile against whatever
the SDK on main happens to expose.
* Java SDK: Clarify task ownership validation
* Java SDK: Simplify manual task registration
Manual interface-based bundles are authored directly by users, so requiring every call to wrap the task class in TaskDef adds noise to the common case while the SDK can retain first-class task definitions internally.
* Fix Java SDK numeric XCom casting
Generate accessors that widen through Number instead: primitive params use
((Number) x).intValue(), and boxed params use Optional.ofNullable(...).orElse(null)
so an absent XCom stays null. Add a java_xcom_casting_example DAG and processor
tests covering int->long->double widening and an absent boxed value.
Signed-off-by: PoAn Yang <payang@apache.org>
* Remove unused key field from Java SDK Builder.XCom annotation
Signed-off-by: PoAn Yang <payang@apache.org>
* Add float cross-type coverage to Java SDK XCom casting tests
Signed-off-by: PoAn Yang <payang@apache.org>
* Remove unused comment
Signed-off-by: PoAn Yang <payang@apache.org>
---------
Signed-off-by: PoAn Yang <payang@apache.org>
* Java SDK: Serialize coordinator client send/receive for thread safety
A Java task may call the SDK client from multiple threads. Every call goes through
one request/response channel to the supervisor. So it must be thread safe.
Signed-off-by: PoAn Yang <payang@apache.org>
* Move requestId outside the lock
Signed-off-by: PoAn Yang <payang@apache.org>
---------
Signed-off-by: PoAn Yang <payang@apache.org>
* Fix Java SDK not retrying failed tasks
* Verify Java SDK UP_FOR_RETRY end-to-end in e2e bundle
Make the Java example bundle's 'load' task fail on its first attempt and
succeed on the retry (retries=1), wire it into the annotation example Dag,
and assert end-to-end that the task passes through UP_FOR_RETRY and ends
success on try_number 2. This exercises the RetryTask path the Java
coordinator now returns.
* Add RetryTask reference in contributing-docs/30_new_language_sdk.rst
* Split load and python_task_2 dep syntax
* Simplify failed-or-retry conditional
* Fix CI error
---------
Co-authored-by: Arnav <a.purohit1903@gmail.com>
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
Jason(Zhe-You) LiuandClaude
PoAn YangandZHE YOU LIU
Tzu-ping Chung
