mirror of
https://github.com/apache/airflow.git
synced 2026-11-04 15:19:48 +00:00
Jason(Zhe-You) Liu
d0b456cdf9
* Add Scala Spark ETL example bundle to the Java SDK e2e tests
## Why
Demonstrate and regression-test that the Java SDK can run a real Scala +
Apache Spark workload, with task logs routed into Airflow via Log4j 2.
## What
- Add `java-sdk/scala_spark_example`: a standalone Scala + Spark 3.5 (local
mode) ETL bundle whose three tasks pass scalar results over XCom and log
through Log4j 2 (`airflow-sdk-log4j2`).
- Run it inside the existing `java_sdk` e2e via a second coordinator and queue
(`scala-jdk` / `scala`) with its own `jars_root`, keeping the Java example
bundle Spark-free.
- Pin the e2e worker JRE to Java 17 and pass Spark's `--add-opens` JVM args.
- Add `TestJavaSDKScalaSparkExample` asserting the tasks succeed and the XComs
match the fixed dataset (5 rows, total revenue 1000).
* Avoid copying jars for docker build
* Verify the Scala Spark transform stage in the Java-SDK e2e test
The e2e test asserted only the extract and load XComs, so the aggregation
stage in the middle of the pipeline could regress without the test noticing.
Assert its XCom as well, drop the unused dataset constants, and note why the
transform reads the upstream count it does not reuse.
* Carry Spark's full Java 17 module options in the Scala Spark e2e test
The Scala Spark coordinator launched the bundle JVM with a hand-curated
subset of Spark's Java 17 module openings. Spark normally injects its full
default set through its own launcher, which the raw JavaCoordinator launch
bypasses. The subset is enough for the toy aggregation but omits openings
that real Spark code paths need (Kryo reflection, off-heap cleaner, charset
decoding, Kerberos), so the example would mislead anyone copying it for a
non-trivial Spark workload. Mirror Spark 3.5.8's full default module option
set instead.
* Standardize Scala Spark example layout and document Spark JVM options
Review feedback on the Scala Spark e2e example asked to make the bundle
source layout consistent -- it mixed the Maven-standard src/main/resources
with flat src/scala and src/resources -- so the whole tree now follows the
standard src/main/{scala,resources} convention. It also asked that the JDK
module options Spark needs under the Java SDK be documented where users can
find them, not only buried in the e2e test, and that comments restating the
code be trimmed.
34 lines
1.1 KiB
Groovy
34 lines
1.1 KiB
Groovy
/*
|
|
* Licensed to the Apache Software Foundation (ASF) under one
|
|
* or more contributor license agreements. See the NOTICE file
|
|
* distributed with this work for additional information
|
|
* regarding copyright ownership. The ASF licenses this file
|
|
* to you under the Apache License, Version 2.0 (the
|
|
* "License"); you may not use this file except in compliance
|
|
* with the License. You may obtain a copy of the License at
|
|
*
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
|
*
|
|
* Unless required by applicable law or agreed to in writing,
|
|
* software distributed under the License is distributed on an
|
|
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
* KIND, either express or implied. See the License for the
|
|
* specific language governing permissions and limitations
|
|
* under the License.
|
|
*/
|
|
|
|
// This is only needed since we want to route the plugin to the local build.
|
|
// You don't need it in your own project.
|
|
pluginManagement {
|
|
repositories {
|
|
mavenLocal()
|
|
gradlePluginPortal()
|
|
}
|
|
}
|
|
|
|
plugins {
|
|
id("org.gradle.toolchains.foojay-resolver-convention") version "0.10.0"
|
|
}
|
|
|
|
rootProject.name = "airflow-java-sdk-scala-spark-example"
|