mirror of
https://github.com/apache/airflow.git
synced 2026-11-04 07:12:48 +00:00
Jason(Zhe-You) Liu
2bda192870
* Java SDK: Record the class that declares each Dag and describe it on request DagDef remembers the outermost class that built it, and a generated @Builder.Dag builder names the annotated class instead of itself. Server accepts --describe-sources <file> to write each Java-declared Dag's declaring class and return without connecting, for the Gradle plugin. * Java SDK: Pack each native Dag's source file into the bundle JAR The packDagSources task runs mainClass in describe mode, finds each Dag's source file through the class file's SourceFile attribute, and packs each unique file with a sources.json mapping under META-INF/airflow/. The JAR manifest names it with Airflow-Java-SDK-Sources. * Java SDK: Bound the describe run and keep its payload to the bundle JAR Every `Jar` task gained a dependency on `packDagSources`, so a sources or javadoc JAR carried the Dag sources and claimed the `Airflow-Java-SDK-Sources` attribute that the Python side picks a bundle JAR out of a directory by. Only `jar` and `shadowJar` carry them now. The describe run had no time bound, so a `main` that does not route through `Server.create(args)` and starts serving instead held the build open with no way out. It now runs under a timeout and falls back to packing the entrypoint's source alone, which is what every other describe failure already does. Say in `serve` and `serveAsync` that a `--describe-sources` server returns without connecting, and record in `sources.json` that its paths are relative to the source directory, which is also where they sit under `META-INF/airflow/sources/`. * Java SDK: Show each Dag's own source and find it outside the package layout The Java importer now passes the Dag ID to the source lookup, so the Code view reads dag_source_paths instead of always showing the entrypoint. The source locator falls back to a unique match on the SourceFile name when the package does not match the directory, using an index built only when the package path misses. An unresolvable entrypoint or Dag class is now logged as a warning. The describe task is no longer up to date after a failed run, so a transient failure does not stick until a source changes. Its hint now points at Server.create(args), which is what answers --describe-sources. Add a shadowJar test, correct the comment on which JARs carry the sources payload, and document that a Dag maps to the class that constructed it. * Java SDK: Use a describe run's output even when the run itself failed * Java SDK: Serve and describe sources from their own Server classes * Java SDK: Pack the sources into the one JAR the bundle is built from * Java SDK: Attach the plugin's documentation to the plugin class * Java SDK: Drop the duplicated classes dir from the describe classpath * Java SDK: Let a bundle set how long the describe run may take
Architectural Decision Records — Language SDKs and coordinators
These ADRs record the cross-cutting architecture decisions behind running non-Python tasks in
Airflow (AIP-108): the coordinator layer, workload
execution, packaging, and how Lang SDKs touch Airflow core surfaces (Dag parsing, DagCode, the
REST API, and the UI). They originated in java-sdk/adr/ and were moved here following the review
discussion on apache/airflow#70059, because they
bind core interfaces and apply to every language SDK, not just the Java SDK.
- ADR-0001: Java SDK / Airflow integration and the coordinator extension point.
- ADR-0002: workload execution through coordinators.
- ADR-0003: pure Java Dags — build-time packaging and code visibility.
- ADR-0004: language-specific Dag file processing.
- ADR-0005: coordinator packaging, module layout, and registration.
- ADR-0006: no Lang-SDK source display for mixed-language (
@task.stub) Dags. - ADR-0007: TaskFlow across the language boundary — argument binding for Lang-SDK tasks.
- ADR-0008: control-flow constructs — grouping, conditional skipping, branching, and triggering a Dag run.
- ADR-0009: provider operators as generated, serialization-only DSL in every Lang SDK.
- ADR-0010: native Dag processing — DagImporter registration and routing.
- ADR-0011: mixed-language Dag processing — task handlers are not Dags.
- ADR-0012: Lang-SDK parse protocol — task handler messages and coordinator verbs.
- ADR-0013: persisted task-handler bindings — resolving Lang-SDK artifacts at parse time.
- ADR-0014: bundle metadata — dropping the Dag inventory, adding a cache digest.
- ADR-0015: per-Dag source in the bundle artifact — de-duplicated by path, with an entrypoint fallback.
Decisions specific to a single SDK stay next to that SDK — for example, the Go SDK's bundle-format
decisions live in go-sdk/adr/. Java-SDK-only interface-design decisions —
the Java call-site syntax itself, rather than anything binding a core surface — live in
java-sdk/adr/, distinct from the cross-cutting ADRs kept here.