6 Commits
Author SHA1 Message Date
Calvin Kirs 77893fb6a4 [fix](filesystem) replace throttled kerberos relogin with proactive TGT refresh to fix flaky GSS failure (#64394)
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #62023

Problem Summary:

The regression case `test_two_hive_kerberos` fails intermittently with:

```
GSSException: No valid credentials provided
  ...
Caused by: javax.security.auth.login.LoginException: Cannot read from System.in
```

Root cause: the `KerberosHadoopAuthenticator` in `fe-filesystem-hdfs`
(introduced in #62023) relies on Hadoop's
`UserGroupInformation.checkTGTAndReloginFromKeytab()`, which has a
hard-coded 60s relogin throttle (`hasSufficientTimeElapsed`). When the
TGT expires inside that throttle window, the relogin is silently
skipped, the subsequent SASL handshake finds no valid ticket, and the
JAAS fallback path tries to prompt on `System.in` — which fails in a
server process.

Fix: replace the throttled reactive relogin with trino's proactive TGT
refresh model (ported 1:1 from trino 435 `KerberosAuthentication` /
`CachingKerberosHadoopAuthentication` / `KerberosTicketUtils`):

- Login eagerly at construction with explicit JAAS options
(`doNotPrompt=true`, `useKeyTab=true`, `storeKey=true`,
`isInitiator=true`) — `doNotPrompt=true` makes the `System.in` prompt
path impossible.
- Compute the next refresh time as ticket start + 80% of ticket
lifetime.
- On each `doAs`, if past the refresh point, perform a fresh keytab
login (no throttle) and swap the new credentials into the existing
`Subject` in place, so the cached UGI stays valid.
- A failed refresh does not advance the refresh time, so the next call
retries immediately; the failure is wrapped as `IOException` to keep the
SPI's checked-exception contract.

The vendored `KerberosTicketUtils` is placed in `fe-foundation`
(`org.apache.doris.foundation.security`) and provided to filesystem
plugins through `fe-filesystem-spi`, which also lets `fe-common`'s
`HadoopKerberosAuthenticator` drop its `io.trino` import.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
- [x] Yes. <!-- Explain the behavior change --> Kerberos keytab relogin
in fe-filesystem-hdfs is now proactive (refresh at 80% of TGT lifetime,
unthrottled) instead of Hadoop's reactive relogin with a 60s throttle.
2026-06-30 10:44:21 +08:00
heguanhui 9f6782e5d0 [fix](build) Fix build and test scripts failing due to missing executable permission (#63330)
### What problem does this PR solve?

Issue Number: close #63225

Problem Summary: Several shell scripts are invoked directly (e.g.,
`./build.sh`) without executable permission, causing "Permission denied"
errors during build, FE UT, BE UT, and broker build processes. This PR
changes all such invocations to use `bash` prefix to ensure they execute
correctly regardless of file permission settings.
2026-06-14 22:35:48 +10:00
Socrates 258c9991de [improvement](build) Optimize FE incremental build packaging (#62915)
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This PR optimizes the Doris FE build path so ordinary `./build.sh --fe`
builds no longer clear `fe-core/target` or rebuild runtime dependency
directories as a side effect.

The main changes are:
- Remove the default `maven-clean-plugin:auto-clean` execution from
`fe-core`, so Maven can reuse `target/classes` for incremental
compilation.
- Replace `maven-dependency-plugin:copy-dependencies` with
`maven-assembly-plugin` and produce `fe/fe-core/target/doris-fe-lib.zip`
as a Maven package artifact.
- Disable recompression of already-compressed runtime dependency jars in
the FE lib assembly to reduce Maven assembly heap pressure.
- Update `build.sh --fe` to run Maven `package` instead of `install`,
then unpack `doris-fe-lib.zip` into `output/fe/lib`.
- Keep `run-fe-ut.sh --coverage` aligned with the new explicit coverage
profile so normal FE UTs do not enable JaCoCo, while coverage runs still
attach the JaCoCo agent and generate reports.
- Remove the build-cache `copy-dependencies` `runAlways` workaround and
align the build-cache config schema with the `1.2.0` schema shipped by
`maven-build-cache-extension:1.2.2`.
- Move non-daily build tasks into explicit profiles: release-only
license/flatten/source artifacts and coverage-only Jacoco setup.
- Keep source artifact generation scoped to the modules that already
generated source artifacts before this change.
- Keep standalone CDC client builds working by installing a flattened
`fe-common` POM before `cdc_client` resolves it outside the FE reactor.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [x] Manual test (add detailed scripts or steps below)
- `mvn -q -Prelease help:effective-pom
-Doutput=/tmp/doris-fe-release-effective-pom.xml -DskipTests
-Dcheckstyle.skip=true -Dskip.doc=true`
- `mvn -q -f fe/pom.xml help:effective-pom
-Doutput=/tmp/doris-fe-effective-pom-schema-1.2.0.xml -DskipTests
-Dcheckstyle.skip=true -Dskip.doc=true`
        - `MAVEN_OPTS="-Xmx8g" DISABLE_BUILD_UI=ON ./build.sh --fe`
        - `DISABLE_BUILD_UI=ON ./build.sh --fe`
- `./run-fe-ut.sh --coverage --run
org.apache.doris.common.PatternMatcherTest`
        - `./fs_brokers/cdc_client/build.sh`
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2026-05-07 14:50:12 +08:00
morrySnow a81c501905 [opt](build) enable maven cache and check maven version (#60819)
### What problem does this PR solve?

Problem Summary:

This pull request updates the Maven build configuration to enforce a
minimum Maven version and improve build performance through caching. The
most important changes are:

Build requirements and enforcement:

* Added the `maven-enforcer-plugin` (version 3.6.2) to the `fe/pom.xml`
build plugins, enforcing that Maven version 3.9.0 or higher is required
to build the project.

Build performance improvements:

* Added the `maven-build-cache-extension` (version 1.2.2) to the
`fe/pom.xml` build extensions, enabling build caching to speed up
repeated builds.
2026-03-01 20:46:16 +08:00
wudi 5f9265695b [Feature](Streaming Job) Extend streaming job to support Postgres synchronization (#59461)
### What problem does this PR solve?

This Issues (https://github.com/apache/doris/issues/58896) implements
multi-table synchronization in MySQL, The main purpose of this PR is to
extend the data source to Postgres.
2026-01-04 14:08:32 +08:00
wudi b7fb2bb178 [Feature](Streaming Job) Extend streaming job to support MySQL synchronization (#58898)
### What problem does this PR solve?

Issue Number: close #58896
2025-12-22 10:45:18 +08:00