* Java SDK: Describe native Java Dags without reference to Python
A Dag authored entirely in Java has no Python in the picture, so teaching
its edge verbs as "Java's spelling of >> and <<" asks the reader to know a
language the feature does not involve. The mixed-language surface keeps its
Python references, because there the Python Dag file is part of what the
reader has to understand.
* Java SDK: Generate Dag and task configuration from the serialization schema
A Java-authored Dag could carry no schedule, retries, tags or any other
Airflow setting, so it could be defined but not configured. Hand-listing
those settings in Java would drift from the Python semantics they mirror,
so they are recorded and validated against Airflow's own Dag serialization
schema instead, and take effect once the runtime answers parse requests.
The `Builder` annotations are generated from a copy of that schema vendored
at java-sdk/sdk/schema/dag-schema.json, so a source release builds without
the monorepo; a prek hook keeps the copy in sync.
The regenerate-java-sdk-verification-metadata hook added in #71869 drops
checksums that no Java SDK build task resolves any more, but the trust
list on main still carried entries for versions superseded by earlier
dependency bumps. The hook therefore rewrites the file in every run, and
static checks fail on every PR regardless of what it touches.
Generated-by: Claude Opus 5
* Java SDK: Draw a Java-authored Dag's edges with before and after
A Dag authored in Java had no way to say what runs after what. Without a
Python Dag file to hold the graph, every task was a root, so a Dag that
Java owns end to end could not be expressed at all.
The verbs are before and after, matching the TypeScript SDK, and leaving
then free for the conditional control flow that reads as a then-branch.
They are the only way to draw an edge, so one graph has one spelling
however it was built.
* Add notes on delibrate implementation choices
---------
Co-authored-by: Tzu-ping Chung <tp@astronomer.io>
* Java SDK: Add setVariable and deleteVariable to the task Client
Signed-off-by: PoAn Yang <payang@apache.org>
* Remove null description case
Signed-off-by: PoAn Yang <payang@apache.org>
---------
Signed-off-by: PoAn Yang <payang@apache.org>
* Java SDK: Register tasks as first-class TaskDef objects
`dag.addTask("extract", Extract.class)` stored tasks as a plain
`Map<String, Class<out Task>>`, which leaves nowhere to hang anything
else a task needs: dependency edges, task-level configuration, and
argument wiring all have to attach to a per-task object, and a map of
classes cannot carry them. Introducing that object now keeps those
follow-ups additive instead of forcing another break of the registration
API later.
The annotation surface keeps `Builder.Dag` / `Builder.Task`, and the
interface users implement keeps the `Task` name, so the definition
objects are `DagDef` and `TaskDef` -- a pairing that stays unambiguous
next to `Task` at a use site. The SDK is pre-1.0, so the old
string-keyed overload is removed outright rather than deprecated.
The KubernetesExecutor lang-SDK example adopts the new names directly rather
than pinning to the old ones: CI now builds those artifacts from the branch
under test (#71527), so the example no longer has to compile against whatever
the SDK on main happens to expose.
* Java SDK: Clarify task ownership validation
* Java SDK: Simplify manual task registration
Manual interface-based bundles are authored directly by users, so requiring every call to wrap the task class in TaskDef adds noise to the common case while the SDK can retain first-class task definitions internally.
* Add Java SDK capability manifest and compatibility matrix
The Language SDK conformance spec defines which TaskInstance states and
capabilities a Language SDK may declare, but nothing lets an SDK say what it
actually supports. Readers of the Java SDK docs cannot tell which parts of the
spec the runtime implements today — and the gaps are real and moving (native Dag
authoring, deferral, and the state stores are not there yet), so an unqualified
"see the conformance spec" overstates what a Java task can do.
The manifest is hand-authored YAML rather than something derived from the SDK's
own sources: it describes the SDK instead of being part of it, so it has no
business shipping in a user's runtime dependencies, and deriving it would mean a
JDK and a Gradle run in a hook that only ever renders a table. It therefore sits
above every subproject in settings.gradle.kts, where no source set can pick it
up. Validation carries the weight a compiler would have: unknown keys are
rejected along with missing ones, since it is the only thing standing between a
typo and a wrong published table.
A prek hook regenerates the contributor-facing README table and the Dokka module
doc from the manifest, so a capability landing in the runtime is a one-line edit
rather than three tables to update by hand. The shared schema, SDK registry, and
renderer let the Go and TypeScript SDKs declare theirs the same way.
* Keep Language SDK compatibility matrices accurate
Contradictory native-Dag declarations could pass validation while being rendered as not applicable. Published matrix tables also need to remain readable without forcing unrelated headers onto one line.
The Java SDK build trusted whatever its configured repositories returned, so a
replaced or shadowed artifact could execute during a build -- including in the
buildSrc code that configures every published module -- without anything
failing. Pinning checksums turns that substitution into a build error, and
declaring repositories in one place gives reviewers a single spot to audit
where bytes may come from.
Checksums alone are trust-on-first-use: they prove an artifact has not changed
since it was recorded, not that it was ever trustworthy. The README therefore
documents the independent cross-check required when regenerating the metadata,
and names the inputs this deliberately does not cover -- the auto-provisioned
JDK toolchain and the downloaded Supervisor Schema -- so the guarantee is not
read as broader than it is.
Serializing each client's whole send → receive → deserialize round trip
behind one lock (#69080) allows only a single request in flight at a
time, so a task issuing concurrent calls over the shared comm socket
pays a full, serialized round trip per call.
Every frame already carries a request id, so a single background
dispatcher can read the socket and route each response by id, with the
lock guarding only the write.
Signed-off-by: PoAn Yang <payang@apache.org>
* Move Lang-SDK ADRs from java-sdk/adr to airflow-core/adr/lang-sdk
* Unwrap ADR list line breaks in Lang-SDK ADR index
Manual line wrapping in the bullet list made the rendered Markdown
harder to scan; one line per entry reads better.
* Add ADR-0006: no Lang-SDK source display for stub DAGs
* Drop the per-Dag source slicing argument from ADR-0006
That point argued against per-Dag filtering of the source view, but that
shape is essentially the upcoming per-file get_source_code on the AIP-85
DagImporter interface, so it is not a reason for this decision and would
read as pre-judging that work.
* Exclude the full Gradle wrapper from the Java SDK source release
Only gradle-wrapper.jar was excluded from the sourceRelease tarball
(LEGAL-570), but gradlew, gradlew.bat, and gradle-wrapper.properties
still shipped, drawing a binding -1 in the 1.0.0-beta1 RC2 vote. The
wrapper stays tracked in git; only the git-archive output used by
sourceRelease now omits it. Document both ways to build an extracted
source package without the wrapper, add a release-verification
checklist, and add a vote-email template with the full 8-artifact
staged list so future RCs don't repeat the RC2 feedback.
* Add LICENSE/NOTICE and reproducible archives to Java SDK jars
Staged convenience-binary jars (main, sources, javadoc, test-fixtures)
were missing META-INF/LICENSE and META-INF/NOTICE, violating ASF
release policy for distributed binaries. Builds were also not
byte-reproducible: jar entries carried wall-clock timestamps, blocking
GitHub-driven publishing. Centralizing both fixes in the
airflow-jvm-conventions plugin, applied by every jar-producing module
(sdk, jul, log4j2, slf4j, jpl, processor, plugin), covers all modules
without per-module duplication.
* Manage airflow-sdk-jpl version in the Java SDK BOM
The BOM constrained sdk, jul, log4j2, processor, and slf4j but not
jpl, even though jpl is published to Nexus and the bundled example
module depends on it. Consumers pulling in jpl via the BOM alone had
no version constraint to resolve against.
* Manage airflow-sdk-gradle-plugin version in the Java SDK BOM
airflow-sdk-gradle-plugin publishes under the same org.apache.airflow
Maven coordinate convention as the other SDK artifacts, but was left
out of the BOM's constraints when airflow-sdk-jpl was added — the same
gap the jpl fix was closing. Consumers importing airflow-sdk-bom still
got no version pin for the plugin artifact.
* Use rc<N> placeholder in Java SDK release verification clone example
* Keep wrapper properties in the archive
* Add verification in BOM to not miss subprojects
* Touch up on Java SDK verification process
---------
Co-authored-by: TP <uranusjr@apache.org>
* Fix Java SDK numeric XCom casting
Generate accessors that widen through Number instead: primitive params use
((Number) x).intValue(), and boxed params use Optional.ofNullable(...).orElse(null)
so an absent XCom stays null. Add a java_xcom_casting_example DAG and processor
tests covering int->long->double widening and an absent boxed value.
Signed-off-by: PoAn Yang <payang@apache.org>
* Remove unused key field from Java SDK Builder.XCom annotation
Signed-off-by: PoAn Yang <payang@apache.org>
* Add float cross-type coverage to Java SDK XCom casting tests
Signed-off-by: PoAn Yang <payang@apache.org>
* Remove unused comment
Signed-off-by: PoAn Yang <payang@apache.org>
---------
Signed-off-by: PoAn Yang <payang@apache.org>
* Java SDK: Serialize coordinator client send/receive for thread safety
A Java task may call the SDK client from multiple threads. Every call goes through
one request/response channel to the supervisor. So it must be thread safe.
Signed-off-by: PoAn Yang <payang@apache.org>
* Move requestId outside the lock
Signed-off-by: PoAn Yang <payang@apache.org>
---------
Signed-off-by: PoAn Yang <payang@apache.org>
* Add Scala Spark ETL example bundle to the Java SDK e2e tests
## Why
Demonstrate and regression-test that the Java SDK can run a real Scala +
Apache Spark workload, with task logs routed into Airflow via Log4j 2.
## What
- Add `java-sdk/scala_spark_example`: a standalone Scala + Spark 3.5 (local
mode) ETL bundle whose three tasks pass scalar results over XCom and log
through Log4j 2 (`airflow-sdk-log4j2`).
- Run it inside the existing `java_sdk` e2e via a second coordinator and queue
(`scala-jdk` / `scala`) with its own `jars_root`, keeping the Java example
bundle Spark-free.
- Pin the e2e worker JRE to Java 17 and pass Spark's `--add-opens` JVM args.
- Add `TestJavaSDKScalaSparkExample` asserting the tasks succeed and the XComs
match the fixed dataset (5 rows, total revenue 1000).
* Avoid copying jars for docker build
* Verify the Scala Spark transform stage in the Java-SDK e2e test
The e2e test asserted only the extract and load XComs, so the aggregation
stage in the middle of the pipeline could regress without the test noticing.
Assert its XCom as well, drop the unused dataset constants, and note why the
transform reads the upstream count it does not reuse.
* Carry Spark's full Java 17 module options in the Scala Spark e2e test
The Scala Spark coordinator launched the bundle JVM with a hand-curated
subset of Spark's Java 17 module openings. Spark normally injects its full
default set through its own launcher, which the raw JavaCoordinator launch
bypasses. The subset is enough for the toy aggregation but omits openings
that real Spark code paths need (Kryo reflection, off-heap cleaner, charset
decoding, Kerberos), so the example would mislead anyone copying it for a
non-trivial Spark workload. Mirror Spark 3.5.8's full default module option
set instead.
* Standardize Scala Spark example layout and document Spark JVM options
Review feedback on the Scala Spark e2e example asked to make the bundle
source layout consistent -- it mixed the Maven-standard src/main/resources
with flat src/scala and src/resources -- so the whole tree now follows the
standard src/main/{scala,resources} convention. It also asked that the JDK
module options Spark needs under the Java SDK be documented where users can
find them, not only buried in the e2e test, and that comments restating the
code be trimmed.
* Java SDK: replace AirflowJulHandler.install() with setup()
install() only added the handler and left JUL's default ConsoleHandler in
place, so every task log record was also written to stderr, which Airflow
captures as task.stderr at ERROR level: each line appeared twice and INFO
records were mislabeled as errors. setup() clears the root logger's handlers
before installing AirflowJulHandler so records flow only through Airflow.
Renamed per review feedback on #68938; the SDK is experimental, so no
deprecation path is needed.
* Java SDK: let AirflowJulHandler.setup() keep existing root handlers
Clearing every root handler is the right default, but it forecloses setups
where a user has intentionally attached their own handler (for example a
FileHandler that mirrors logs to disk) that does not conflict with Airflow's
task-log and task.stderr capture. Raised in review on #68986. A clean=true
default preserves the de-duplicating behaviour while clean=false adds the
AirflowJulHandler alongside the existing handlers. Both paths stay idempotent
so repeated calls never install more than one AirflowJulHandler.
* Fix Java SDK not retrying failed tasks
* Verify Java SDK UP_FOR_RETRY end-to-end in e2e bundle
Make the Java example bundle's 'load' task fail on its first attempt and
succeed on the retry (retries=1), wire it into the annotation example Dag,
and assert end-to-end that the task passes through UP_FOR_RETRY and ends
success on try_number 2. This exercises the RetryTask path the Java
coordinator now returns.
* Add RetryTask reference in contributing-docs/30_new_language_sdk.rst
* Split load and python_task_2 dep syntax
* Simplify failed-or-retry conditional
* Fix CI error
---------
Co-authored-by: Arnav <a.purohit1903@gmail.com>
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
The supervisor's task routing (added in #65958) reads queue, pool_slots and
priority_weight from the TaskInstance it receives, but those fields lived in
a TaskInstanceDTO duplicated between airflow-core and task-sdk and kept in
sync by an AST-comparison prek hook, outside the versioned execution API
schema the Task SDK datamodels are generated from.
Add the three fields to the execution API TaskInstance schema (with a Cadwyn
version change) and regenerate the SDK datamodels, so the generated
TaskInstance carries everything the supervisor needs. Re-point StartupDetails
and the coordinator interfaces at the generated model, fold the core
TaskInstanceDTO into a subclass of the schema model that only adds the
executor-side fields, and drop the duplicated task-sdk DTO and the sync hook.
The unused parent_context_carrier field (no readers or writers) is removed.
* Keep pool_slots and priority_weight executor-side; only queue joins the schema
The supervisor only routes on queue; pool_slots and priority_weight are
executor concerns (queued-workload priority ordering and edge concurrency
slot accounting) that the worker never reads. Keep them on the executor
DTO, serialized as before, and add only queue to the worker-facing schema.
The generated TaskInstance no longer accepts pool_slots/priority_weight
kwargs, its queue is Optional in the generated typing, and the DTO now
validates the inherited hostname field that the ECS adoption test's mock
did not set.
Jason(Zhe-You) Liu
Jarek Potiuk
PoAn Yang
dependabot[bot]andTzu-ping Chung
Tzu-ping Chung
Andrew Chang
Kaxil Naik