mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
bingquanzhao
62dc78fbb0
Replace HttpClient5 async with HttpClient4 sync to fix
CircularRedirectException (1.2.0 -> 1.2.1)
The logstash-output-doris plugin uses Apache HttpClient5 async client to
PUT stream load requests. Against SelectDB Cloud / BYOC FE — which
returns '307 + Connection: close' on stream load — the async client
fails with CircularRedirectException under any meaningful concurrency /
body size.
Root cause:
1. HC5 async does not strictly block body transmission while waiting for
'100 Continue'. When FE returns 307 before issuing 100, the entity
producer has already started writing; FE closing the connection then
yields an IOException mid-transfer.
2. HC5 default exec chain wraps RedirectExec around
AsyncHttpRequestRetryExec. The recoverable IOException triggers an
internal retry that re-enters the same FE -> 307 path, but
RedirectLocations from the first attempt is still populated, so the same
BE URL is detected as 'already visited' and reported as a circular
redirect.
This is a real HC5-vs-HC4 implementation difference, not a configuration
issue. The Doris Flink connector also follows FE 307 to BE in its
default path (autoRedirect=true) and works correctly precisely because
it uses HC4 sync: HC4 honors 'Expect: 100-continue' strictly, so when FE
307s without sending 100, the entity is left unconsumed and HC4's
RedirectExec follows the redirect normally.
This patch aligns the plugin with the Flink connector's HTTP layer:
- bump gem version 1.2.0 -> 1.2.1
- httpclient5 5.4.2 (async) -> httpclient 4.5.13 (sync)
- SimpleRequestBuilder -> HttpPut + ByteArrayEntity (repeatable)
- HttpAsyncClients defaults -> HttpClients with:
* setRequestExecutor(HttpRequestExecutor(60s))
* setRedirectStrategy(DorisRedirectStrategy) (isRedirectable=true,
strip userinfo, normalize empty query)
* setRetryHandler(DefaultHttpRequestRetryHandler(0, false))
* setConnectionReuseStrategy(NoConnectionReuseStrategy.INSTANCE)
* RequestConfig.setExpectContinueEnabled(true)
- Async future plumbing in TableEvents replaced with sync
response_code / response_body / response_error fields.
- Stringify both key and value at request.addHeader call site: HC4's
addHeader(String, String) is strict on types whereas HC5 had a
permissive (String, Object) overload; user configs commonly carry
Float / Integer values like 'max_filter_ratio => 1.0'.
- Drop 's.requirements << jar ...' from gemspec: with JARs vendored
under
lib/, the maven lookup at install time is unnecessary and forced users
to set JARS_SKIP=true for offline installs.
Pipeline configuration, retry queue, save_on_failure, group_commit,
label generation, header handling - all unchanged.
Verified on a SelectDB BYOC cluster mirroring the reported production
shape (16 workers x 10000 batch x 200,000 events):
- Before: 100% requests fail with CircularRedirectException
- After: 20/20 stream loads Status=Success, 200,000/200,000 rows
ingested, 0 HTTP-layer errors.
logstash-output-doris
A Logstash output plugin that ships events to Doris via stream load.
Docs: https://doris.apache.org/docs/dev/ecosystem/logstash
Build
Prerequisites: JRuby (>= 9.4 with Java 21, or 9.2 with Java 8/11) and the
jar-dependencies gem (jruby -S gem install jar-dependencies).
# 1. Vendor HttpClient4 + transitive jars into lib/ and generate the loader.
# Reads the 'jar' entry from logstash-output-doris.gemspec.
jruby -e "require 'jars/installer'; Jars::Installer.new.vendor_jars"
# 2. Build the gem.
jruby -S gem build logstash-output-doris.gemspec
Produces logstash-output-doris-<version>-java.gem.
Install
The jars are already vendored inside the gem, so the install hook does not
need to talk to Maven — pass JARS_SKIP=true to skip the lookup:
JARS_SKIP=true $LS_HOME/bin/logstash-plugin install \
logstash-output-doris-<version>-java.gem
Air-gapped install (offline pack)
For a Logstash host without internet access, build an offline pack on a connected host first:
# On a connected host (same Logstash major version as the target):
JARS_SKIP=true $LS_HOME/bin/logstash-plugin install \
logstash-output-doris-<version>-java.gem
$LS_HOME/bin/logstash-plugin prepare-offline-pack \
--output logstash-output-doris-<version>-offline-pack.zip \
logstash-output-doris
# Then on the air-gapped target:
$LS_HOME/bin/logstash-plugin install \
file:///path/to/logstash-output-doris-<version>-offline-pack.zip