nprasad2077/airflow
1
0 Fork 0
mirror of https://github.com/apache/airflow.git synced 2026-11-04 07:12:48 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
airflow/.rat-excludes
T
Jason(Zhe-You) Liu 12bb6d9b95 Cover native TypeScript and Java Dags, including control flow, in the e2e suites (#73847) 
* TS SDK: cover native TypeScript Dags end to end

* Parse the native TypeScript Dag in the compose e2e

Put the example bundle in the Dag bundle as well, add a root-less
"ts-native" Node coordinator that claims it there, and give the Dag
processor node. The existing "ts" coordinator still runs every TypeScript
task, so the native Dag test needs no gate anymore.

* Check the native TypeScript Dag's source and triggered run

The Code view must show the bundle's entry module, and
trigger_downstream, run by the TypeScript runtime, must trigger
typescript_example with its conf and link to that run.

* Keep the containerized Java build's Maven repository on the host

The JVM takes user.home from /etc/passwd, so when the host UID is one the
JDK image defines (1000 is its "ubuntu" user), publishToMavenLocal wrote
into the container and the example builds could not find the SDK. Name the
mounted repository with maven.repo.local instead of relying on HOME.

* Add a native Java Dag to the compose e2e

A dedicated bundle declares two Dags in Java, one with the interface API
and one with annotations, and routes every task to a "java-native" queue.
Its JAR goes into the Dag bundle, where a root-less Java coordinator parses
it in the Dag processor and runs its tasks. The test checks the edges, the
queues, the run, the XComs and the Dag source the JAR embeds.

* Run the Lang-SDK e2e jobs when native Dag parsing changes

The Java and TypeScript e2e suites now parse native Dags through the Dag
processor, the Dag bag and the coordinators' Dag importers, so changes to
those sources select both jobs. The Java native-Dag bundle selects the
Java job.

* Share the Dag run lookup and the native Java queue name

wait_for_dag_run now polls through get_dag_run, and the annotation Dag
takes its queue from the interface Dag's constant.

* Build the k8s TypeScript artifacts only for the native Dag test

setup-lang-sdk-test packed the TypeScript bundle and built its worker image
for every lang-SDK run, so the Go and Java k8s job, which provisions no Node
toolchain, failed during setup. Build and upload them only with
--ts-sdk-native-dag-test (RUN_TS_SDK_NATIVE_DAG_K8S_TESTS), the variable the
native TypeScript k8s test is gated on.

* Key the Lang-SDK e2e jobs on native Dag modules only

Drop the shared Dag processing files (dagbag, manager, processor)
from the trigger list, since most of their changes are off the
Lang-SDK path and the canary run still covers them. Add
serialized_objects.py, which holds the checks every native Dag goes
through.

* Build the TypeScript SDK before packing the k8s example bundle

The example's airflow-ts-pack bin points at the SDK's dist/, so on a
clean checkout pnpm skipped the bin link and the pack failed. Follow
the compose e2e sequence on both the host and container paths, and
let the container user enable corepack under a writable HOME. The
docstring no longer claims CI provisions Node for this test.

* Pack the TypeScript example against the built SDK

The example inherited the root tsconfig's paths alias, so esbuild
bundled the SDK from ../src instead of the shipped dist/, and the
e2e tests stopped covering the published build. Clearing paths in
the example also fixes its TS6059 typecheck errors.

* Explain why the k8s native TypeScript Dag test is skipped

The Dag processor already dispatches parse requests, so name the
real gaps instead: the bundle reaches only worker pods, the ts-sdk
coordinator is limited to that bundle, and the Dag processor image
has no Node. Also fix the gating env var the README named.

* Document how the TypeScript example's native Dag gets parsed

Say the Dag processor needs node to parse the native Dag, link to
the parsing docs, and show how to trigger it.

* Name the skipped-branch e2e test after what it checks

The test asserts the skipmixin XCom, not a clear, so the old name
read as if the clear path were covered.

* Parse native Dags in the e2e by class, not by Dag bundle name

Java: java-native loses dag_bundle_name and only parses, picked for the
Dags folder by [sdk] dag_bundle_to_coordinator among the four
JavaCoordinators. Its queue routes to java-jdk, so native tasks run on the
coordinator the queue names.

TypeScript: ts is the only NodeCoordinator, so it parses the native Dag
without the ts-native coordinator. The artifact bundles keep their
.airflowignore so they are not parsed as native Dag files.

* Drop the dag_bundle_name reason from the k8s native TypeScript skip

dag_bundle_name no longer limits which Dag bundle is parsed. The other two
reasons still apply.

* Say which coordinator runs the native e2e Java tasks

The java-native queue routes to the java-jdk coordinator, and the
java-native coordinator only parses the JAR.

* Read a native Dag example's served Dags with bundleDags

The test called listBundleNativeDags, which main's TypeScript SDK does not
have. bundleDags returns the same Dags, keyed by dag_id.

* Keep the k8s lang-SDK values comment within the YAML line limit

Adding ts-task-handlers to the comment that lists the task handler Dag
bundles made its first line 114 characters long, over the 110 that
yamllint allows.

* Rework the lang-SDK k8s harness for mixed-language and native Dag tests

Renames the combined Go + Java system test to mixed-language and adds
TypeScript to it, adds one native Dag test per language SDK (Java and
TypeScript; Go is skipped, native Dag parsing has no Go support yet),
and drops the TypeScript-only gate: every lang-SDK artifact now builds
unconditionally.

Every coordinator queue now shares one runtime image (prod + JRE +
Node) and one worker pod template. A coordinator downloads its own
task_handler_bundle_name bundle directly from S3, the same way any
Python Dag bundle works, so the init container and stage_artifacts.py
staging step are gone -- restoring the Go binary's execute bit after
an S3 download is now ExecutableCoordinator's own job (see the
executable-bundle-exec-bit PR this depends on). Every task-handler
bucket carries an .airflowignore so the Dag processor never tries to
parse a packed artifact as a Dag; the two native buckets carry none,
since parsing them is the point. With one coordinator per language,
no [sdk] dag_bundle_to_coordinator entry is needed.

Also updates the k8s CI job to provision Node, and the breeze docs and
unit tests to match.

* Exclude airflowignore_all from the license check

It's a plain .airflowignore-style pattern file, not source, same as
the existing .airflowignore entry.

* Cover If, Switch, a task group and TriggerDagRun in the native Java Dag e2e

Both java_native_e2e (interface API) and java_native_annotation_e2e
(annotations) now also wire a task group, an If, a Switch and a
trigger_downstream task onto their extract -> transform -> load chain,
against a new third Dag, java_native_target_e2e, that the trigger starts.

The Code view test now expects each Dag's own declaring source file
rather than always the bundle's main class, since the bundle embeds one
source per Dag.


* Cover the same native Java Dag features in the k8s native Dag test

TestNativeJavaDagOnKubernetes now follows java_native_e2e's task group,
If, Switch and trigger_downstream, un-pausing java_native_target_e2e
first since trigger_downstream waits for the run it starts.


* Fix the native TypeScript example's trigger to the current Dag.task API

dag.triggerDagRun(spec) is not a method; a trigger is a task built by
the standalone triggerDagRun(spec) and placed with dag.task(taskId,
trigger). dag.if and dag.switch likewise take the deciding handler
itself and place it, not an already-placed TaskRef, so has_rows and
pick_cadence now pass their handler straight to dag.if/dag.switch with
an explicit taskId.

Also use datetime.UTC instead of the deprecated timezone.utc alias.

* Remove client methods a rebase onto main duplicated

Rebasing onto a main that had independently grown set_variable,
get_tasks, get_dag_source and get_dag_run let both copies through as a
clean, silent merge. Keep one of each.

* Fix the TS Code view test for per-Dag source

The bundle now embeds one source file per Dag, so the Code view shows
native.ts, the file that actually declares typescript_native_example,
not main.ts's Bundle registration.

* Fix the Java trigger test's XCom key

trigger_downstream pushes the triggered run's ID under the
trigger_run_id key, not return_value, matching the TS and Go runtimes.


* Add the /coordinator and /hitl subpath aliases typecheck now needs

tsconfig.json's path alias only covered the bare package import.
Including example/src in typecheck (for the native Dag work) now also
compiles main's own HITL example files, which import
apache-airflow-ts-sdk/hitl; add that and the /coordinator subpath to
both tsconfig.json and vitest.config.ts so the compiler and the runner
keep agreeing.

Also follow TaskBody's trigger -> operator rename in the native Dag
example test.

---------
2026-10-11 10:34:40 +01:00

346 lines
5.4 KiB
Plaintext
Raw Permalink Blame History

# Note: these patterns are applied to single files or directories, not full paths
# coverage/* will ignore any coverage dir, but airflow/www/static/coverage/* will match nothing
.git-blame-ignore-revs
.github/*
.gitignore
.gitattributes
.gitrepo
.airflow_db_initialised
.airflowignore
.babelrc
.coverage
.codecov.yml
.codespellignorelines
.eslintignore
.eslintrc
.prettierignore
.prettierrc
.rat-excludes
.stylelintignore
.stylelintrc
.env
.venv
requirements
requirements.txt
.*log
.travis.yml
.*pyc
.*lock
dist
build
airflow.egg-info
apache_airflow.egg-info
.idea
metastore_db
.*sql
.*svg
.*csv
.*md5
.*zip
.*lock
logs
.bash_aliases
venv
clients/*
files/*
dags/*
generated/*
.gitmodules
prod_image_installed_providers.txt
airflow_pre_installed_providers.txt
Dockerfile.pmc
# Generated doc files
.*html
_build/*
_static/*
_images/*
.buildinfo
searchindex.js
_api/*
# the word list for checking spellings
spelling_wordlist.txt
# Apache Rat does not detect BSD-2 clause properly
# it is compatible according to http://www.apache.org/legal/resolved.html#category-a
kerberos_auth.py
airflow_api_auth_backend_kerberos_auth_py.html
3rd-party-licenses/*
parallel.js
underscore.js
jquery.dataTables.min.js
jqClock.min.js
dagre-d3.min.js
d3.v3.min.js
ace.js
node_modules/*
.*json
coverage/*
git_version
flake8_diff.sh
coverage*.xml
_sources/*
robots.txt
rat-results.txt
apache-airflow-.*\+source.tar.gz.*
apache-airflow-.*\+bin.tar.gz.*
PULL_REQUEST_TEMPLATE.md
PROVIDER_CHANGES*.md
manifests/*
redirects.txt
reproducible_build.yaml
# Locally mounted files
.*egg-info/*
.bash_history
.bash_aliases
.inputrc
# the example notebook is ASF 2 licensed but RAT cannot read this
input_notebook.ipynb
# .git might be a file in case of worktree
.git
tmp
# Vendored-in code
.*_vendor
# generated doc-only-changes.txt
.latest-doc-only-change.txt
# Chart ignored files
chart/.gitignore
chart/values.schema.json
chart/Chart.lock
chart/values_schema.schema.json
chart/Chart.yaml
# Generated autocomplete files
./dev/breeze/autocomplete/*
# Generated devel_deps files
devel_deps.txt
# Newsfragments are snippets that will be, eventually, consumed into RELEASE_NOTES
newsfragments/*
# Warning file generated
warnings.txt
warn-summary-*.txt
# Dev stuff
tests/*
scripts/*
images/*
dev/*
**/*.iml
out/*
airflow-build-dockerfile*
# Sha files
.*sha256
# DOAP file
doap_airflow.rdf
# pyenv
.python-version
# nvm (Node Version Manager)
.nvmrc
# PKG-INFO file
PKG-INFO
# checksum files
.*\.md5sum
last-eval-hash.txt
# Openapi files
.openapi-generator-ignore
version.txt
v2*.yaml
_private_ui*.yaml
# Front end generated files
api-generated.ts
openapi-gen
pnpm-lock.yaml
# python generated file
generated.py
auth_generated.py
# hash files
www-hash.txt
# go setup files
**/go.mod
**/go.sum
**/protov1/*
# go mocks
**/mocks/*
# Generated protobuf files
.*proto
.*pb.go
.*_grpc.pb.go
# Kubernetes env
.env
# SVG files
**/*.svg
# Doc only change marker file
**/.latest-doc-only-change.txt
**/*-gen/*
# Redirects
**/redirects.txt
# Ignore files
**/.git-blame-ignore-revs
**/.gitattributes
**/.rat-excludes
**/.gitignore
**/.prettierignore
**/.prettierrc
**/.airflowignore
**/.airflowignore_glob
**/airflowignore_all
# Vendor includes
**/_vendor/
# Generated files
**/*-generated.yaml
**/*-generated.py
**/generated.py
**/generated/*
**/auth_generated.py
# Lock files
**/pnpm-lock.yaml
**/yarn.lock
**/Chart.lock
**/uv.lock
# Generated UI files
**/ui/index.html
**/ui/dev/index.html
**/ui/dist/index.html
**/_private_ui.yaml
**/dist/**
**/www/index.html
# Nunjucks template files (frontmatter must be at line 1, no room for license header)
**/*.njk
# PNG files
**/*.png
# CSV files
**/*.csv
# LICENCE files
**/LICENCE*.txt
**/LICENSE*.txt
# Checksum files
**/*.sha256
**/*.md5sum
# Requirement files
**/requirements.txt
# Hashes
**/command_hashes.txt
**/www-hash.txt
# Spelling wordlist
**/spelling_wordlist.txt
**/dictionary.txt
# Empty files
**/empty.txt
# Script files
**/script
**/script.bteq
**/script_utf16.bteq
# Reproducible build files
**/reproducible_build.yaml
# Other files
**/test_notifier.txt
**/email.html
**/*.log
**/example_upload.txt
**/dummy.pdf
**/java_streaming_src/*
**/kube_config
**/prod_image_installed_providers.txt
**/text.txt
**/newsfragments/**
**/warnings.txt
**/rtd-deprecation/404.html
**/.env
**/*.jsonl
**/tdload_src_file.txt
# API files
**/_api/**
**/node_modules/**
# Doc files
/docs/.latest-doc-only-change.txt
/docs/redirects.txt
/docs/integration-logos/*.svg
/docs/img/*.md5sum
/docs/img/*.svg
# Log files
*.log
# md5 sum files
.*\.md5sum
# Generated files
*generated.*
/src/airflow/providers/keycloak/auth_manager/openapi/v2-keycloak-auth-manager-generated.yaml
/src/airflow/providers/edge3/plugins/www/*
/src/airflow/providers/edge3/worker_api/v2-edge-generated.yaml
/src/airflow/providers/fab/auth_manager/api_fastapi/openapi/v2-fab-auth-manager-generated.yaml
/src/airflow/providers/fab/www/static/dist/*
/any/dag_id=dag_for_testing_redis_task_handler/run_id=test/task_id=task_for_testing_redis_log_handler/attempt=1.log
/src/airflow/providers/google/ads/.gitignore
# Vendored-in code
/src/airflow/providers/google/_vendor/*
# TypeScript bundle golden fixture: its bytes are digest-pinned, so no license header can be added
/ts-sdk/tests/cli/fixtures/bundle-v1.min.mjs
# Java SDK build outputs
/java-sdk/bin/*
/java-sdk/build/*
/java-sdk/*/bin/*
/java-sdk/*/build/*
# Git ignore file
.gitignore
# Last release date is automatically generated
**/.last_release_date.txt
Reference in new issue View Git Blame Copy Permalink