* Move Lang-SDK ADRs from java-sdk/adr to airflow-core/adr/lang-sdk
* Unwrap ADR list line breaks in Lang-SDK ADR index
Manual line wrapping in the bullet list made the rendered Markdown
harder to scan; one line per entry reads better.
* Add ADR-0006: no Lang-SDK source display for stub DAGs
* Drop the per-Dag source slicing argument from ADR-0006
That point argued against per-Dag filtering of the source view, but that
shape is essentially the upcoming per-file get_source_code on the AIP-85
DagImporter interface, so it is not a reason for this decision and would
read as pre-judging that work.
* Exclude the full Gradle wrapper from the Java SDK source release
Only gradle-wrapper.jar was excluded from the sourceRelease tarball
(LEGAL-570), but gradlew, gradlew.bat, and gradle-wrapper.properties
still shipped, drawing a binding -1 in the 1.0.0-beta1 RC2 vote. The
wrapper stays tracked in git; only the git-archive output used by
sourceRelease now omits it. Document both ways to build an extracted
source package without the wrapper, add a release-verification
checklist, and add a vote-email template with the full 8-artifact
staged list so future RCs don't repeat the RC2 feedback.
* Add LICENSE/NOTICE and reproducible archives to Java SDK jars
Staged convenience-binary jars (main, sources, javadoc, test-fixtures)
were missing META-INF/LICENSE and META-INF/NOTICE, violating ASF
release policy for distributed binaries. Builds were also not
byte-reproducible: jar entries carried wall-clock timestamps, blocking
GitHub-driven publishing. Centralizing both fixes in the
airflow-jvm-conventions plugin, applied by every jar-producing module
(sdk, jul, log4j2, slf4j, jpl, processor, plugin), covers all modules
without per-module duplication.
* Manage airflow-sdk-jpl version in the Java SDK BOM
The BOM constrained sdk, jul, log4j2, processor, and slf4j but not
jpl, even though jpl is published to Nexus and the bundled example
module depends on it. Consumers pulling in jpl via the BOM alone had
no version constraint to resolve against.
* Manage airflow-sdk-gradle-plugin version in the Java SDK BOM
airflow-sdk-gradle-plugin publishes under the same org.apache.airflow
Maven coordinate convention as the other SDK artifacts, but was left
out of the BOM's constraints when airflow-sdk-jpl was added — the same
gap the jpl fix was closing. Consumers importing airflow-sdk-bom still
got no version pin for the plugin artifact.
* Use rc<N> placeholder in Java SDK release verification clone example
* Keep wrapper properties in the archive
* Add verification in BOM to not miss subprojects
* Touch up on Java SDK verification process
---------
Co-authored-by: TP <uranusjr@apache.org>
* Fix Java SDK numeric XCom casting
Generate accessors that widen through Number instead: primitive params use
((Number) x).intValue(), and boxed params use Optional.ofNullable(...).orElse(null)
so an absent XCom stays null. Add a java_xcom_casting_example DAG and processor
tests covering int->long->double widening and an absent boxed value.
Signed-off-by: PoAn Yang <payang@apache.org>
* Remove unused key field from Java SDK Builder.XCom annotation
Signed-off-by: PoAn Yang <payang@apache.org>
* Add float cross-type coverage to Java SDK XCom casting tests
Signed-off-by: PoAn Yang <payang@apache.org>
* Remove unused comment
Signed-off-by: PoAn Yang <payang@apache.org>
---------
Signed-off-by: PoAn Yang <payang@apache.org>
* Java SDK: Serialize coordinator client send/receive for thread safety
A Java task may call the SDK client from multiple threads. Every call goes through
one request/response channel to the supervisor. So it must be thread safe.
Signed-off-by: PoAn Yang <payang@apache.org>
* Move requestId outside the lock
Signed-off-by: PoAn Yang <payang@apache.org>
---------
Signed-off-by: PoAn Yang <payang@apache.org>
* Add Scala Spark ETL example bundle to the Java SDK e2e tests
## Why
Demonstrate and regression-test that the Java SDK can run a real Scala +
Apache Spark workload, with task logs routed into Airflow via Log4j 2.
## What
- Add `java-sdk/scala_spark_example`: a standalone Scala + Spark 3.5 (local
mode) ETL bundle whose three tasks pass scalar results over XCom and log
through Log4j 2 (`airflow-sdk-log4j2`).
- Run it inside the existing `java_sdk` e2e via a second coordinator and queue
(`scala-jdk` / `scala`) with its own `jars_root`, keeping the Java example
bundle Spark-free.
- Pin the e2e worker JRE to Java 17 and pass Spark's `--add-opens` JVM args.
- Add `TestJavaSDKScalaSparkExample` asserting the tasks succeed and the XComs
match the fixed dataset (5 rows, total revenue 1000).
* Avoid copying jars for docker build
* Verify the Scala Spark transform stage in the Java-SDK e2e test
The e2e test asserted only the extract and load XComs, so the aggregation
stage in the middle of the pipeline could regress without the test noticing.
Assert its XCom as well, drop the unused dataset constants, and note why the
transform reads the upstream count it does not reuse.
* Carry Spark's full Java 17 module options in the Scala Spark e2e test
The Scala Spark coordinator launched the bundle JVM with a hand-curated
subset of Spark's Java 17 module openings. Spark normally injects its full
default set through its own launcher, which the raw JavaCoordinator launch
bypasses. The subset is enough for the toy aggregation but omits openings
that real Spark code paths need (Kryo reflection, off-heap cleaner, charset
decoding, Kerberos), so the example would mislead anyone copying it for a
non-trivial Spark workload. Mirror Spark 3.5.8's full default module option
set instead.
* Standardize Scala Spark example layout and document Spark JVM options
Review feedback on the Scala Spark e2e example asked to make the bundle
source layout consistent -- it mixed the Maven-standard src/main/resources
with flat src/scala and src/resources -- so the whole tree now follows the
standard src/main/{scala,resources} convention. It also asked that the JDK
module options Spark needs under the Java SDK be documented where users can
find them, not only buried in the e2e test, and that comments restating the
code be trimmed.
* Java SDK: replace AirflowJulHandler.install() with setup()
install() only added the handler and left JUL's default ConsoleHandler in
place, so every task log record was also written to stderr, which Airflow
captures as task.stderr at ERROR level: each line appeared twice and INFO
records were mislabeled as errors. setup() clears the root logger's handlers
before installing AirflowJulHandler so records flow only through Airflow.
Renamed per review feedback on #68938; the SDK is experimental, so no
deprecation path is needed.
* Java SDK: let AirflowJulHandler.setup() keep existing root handlers
Clearing every root handler is the right default, but it forecloses setups
where a user has intentionally attached their own handler (for example a
FileHandler that mirrors logs to disk) that does not conflict with Airflow's
task-log and task.stderr capture. Raised in review on #68986. A clean=true
default preserves the de-duplicating behaviour while clean=false adds the
AirflowJulHandler alongside the existing handlers. Both paths stay idempotent
so repeated calls never install more than one AirflowJulHandler.
* Fix Java SDK not retrying failed tasks
* Verify Java SDK UP_FOR_RETRY end-to-end in e2e bundle
Make the Java example bundle's 'load' task fail on its first attempt and
succeed on the retry (retries=1), wire it into the annotation example Dag,
and assert end-to-end that the task passes through UP_FOR_RETRY and ends
success on try_number 2. This exercises the RetryTask path the Java
coordinator now returns.
* Add RetryTask reference in contributing-docs/30_new_language_sdk.rst
* Split load and python_task_2 dep syntax
* Simplify failed-or-retry conditional
* Fix CI error
---------
Co-authored-by: Arnav <a.purohit1903@gmail.com>
Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
The supervisor's task routing (added in #65958) reads queue, pool_slots and
priority_weight from the TaskInstance it receives, but those fields lived in
a TaskInstanceDTO duplicated between airflow-core and task-sdk and kept in
sync by an AST-comparison prek hook, outside the versioned execution API
schema the Task SDK datamodels are generated from.
Add the three fields to the execution API TaskInstance schema (with a Cadwyn
version change) and regenerate the SDK datamodels, so the generated
TaskInstance carries everything the supervisor needs. Re-point StartupDetails
and the coordinator interfaces at the generated model, fold the core
TaskInstanceDTO into a subclass of the schema model that only adds the
executor-side fields, and drop the duplicated task-sdk DTO and the sync hook.
The unused parent_context_carrier field (no readers or writers) is removed.
* Keep pool_slots and priority_weight executor-side; only queue joins the schema
The supervisor only routes on queue; pool_slots and priority_weight are
executor concerns (queued-workload priority ordering and edge concurrency
slot accounting) that the worker never reads. Keep them on the executor
DTO, serialized as before, and add only queue to the worker-facing schema.
The generated TaskInstance no longer accepts pool_slots/priority_weight
kwargs, its queue is Optional in the generated typing, and the DTO now
validates the inherited hostname field that the ECS adoption test's mock
did not set.
Jason(Zhe-You) Liu
PoAn Yang
Andrew Chang
Tzu-ping Chung
Kaxil Naik
