mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
Gabriel
c7d1df40c0
### What problem does this PR solve?
Elasticsearch scans can return values that fail SQL predicates when
indexed values differ from the values Doris reads, or when a scan
switches all selected columns from docvalues to source. Parallel alias
scans can also bypass alias filters/routing, and supported mapping
shapes can fail during planning.
For example, projecting a normalized keyword forces a source read for
its siblings. A numeric alias then reads NULL although its docvalue is
5; a declared empty ARRAY remains nonnull although ES considers it
missing; a date-only string reads local midnight instead of the indexed
UTC midnight. Boolean empty strings and coerced integer strings must
also remain readable: `""` means false and `"1.2e3"` means 1200. Keeping
their indexed predicates while pruning SQL residuals returns incorrect
rows.
#### Execution flow
1. Select a populated mapping independently of alias response order.
Distinguish typeless flags/root objects from legacy type wrappers,
including a legacy type named `runtime`, and preserve array annotations.
Accept empty/configuration-only alias backings during safety analysis
while retaining standalone schema diagnostics.
2. Resolve the selected keyword query path and docvalue read path
separately on every backing, including filter-only columns. Fetch index
mapping defaults with the existing metadata request and honor per-field
overrides for every alias backing; inherited `ignore_malformed` can
remove indexed values while retaining source values. Ignore
transformations on unused sibling paths and unrelated source filters.
Missing/read-changing paths disable the affected docvalue entry. Root
`enabled:false` and disabled source (`enabled:false` or `mode:disabled`)
restrict every affected column.
3. Determine the final scan-wide read mode before building DSL. In
source mode, STRING/ARRAY predicates and predicates on rounded numbers,
aliases, scripted fields and default dates retain local residuals
without narrowing the ES query. Eligible pure docvalue projections
preserve their indexed-value behavior; scalar `_id` metadata remains
pushable.
4. Keep searchability separate from value equivalence: supported
doc-value-only fields remain queryable with `index=false`. Require
existence for SQL inequalities. Normalize retained equality negation,
double negation and Boolean negation inside the connector. Retained
ranges keep the original ES query complement with an existence guard: an
inverse existential range changes matching on arrays such as [1,2]. This
preserves NULL rejection when optimizer NOT simplification is disabled.
5. Pass residual tracking and the existing early-LIMIT restrictions
through the current scan contract. Native `esquery` remains remote-only:
native-only expressions and independent AND conjuncts work; an
indivisible Boolean expression requiring local native evaluation fails
explicitly during planning.
6. Use both readers' shared parser for supported source representations.
Boolean empty strings become false in scalar and ARRAY paths. Scalar
numeric/Boolean/date singleton source arrays read their element; empty
and null-only arrays read NULL, while STRING/JSONB and declared ARRAY
containers remain intact. JSONB singleton, empty and null-element arrays
retain their original JSON container. Declared ARRAYs preserve nullable
members; numeric empty strings become SQL NULL in scalar and ARRAY
paths, while Boolean empty strings remain false. Integer coercion
follows the ES family: byte/short/integer strings use double rounding
before truncation, while long strings retain exact decimal/exponent
handling and 64-bit precision. Bounds are checked before narrowing.
Declared integer ARRAY elements share the family-specific conversion.
Numeric floating tokens are converted directly with checked bounds,
preserving numeric INT64_MIN and the Java upper-endpoint conversion.
Floating strings and narrow integer strings accept Java hexadecimal
exponents, type suffixes and outer ASCII whitespace; FLOAT strings round
directly to float rather than through double. Epoch-millis source dates
accept zero, short/signed decimal strings and fractional numeric values,
with checked bounds and second-precision floor semantics. Four-digit
default-format years retain their calendar interpretation; pre-epoch
formatted docvalues use the same floor.
7. For a source read involving disabled-source backings, pass an
optional `read_full_source` flag through the existing ES property map.
BE requests `_source:true`: disabled hits have missing values, while
enabled alias backings retain their source. Ordinary source scans keep
field projection and pure docvalue scans keep docvalue requests.
8. Scan aliases once by their original name so ES applies alias filters
and search routing without duplicate shared-shard scans. Physical
indices retain shard-parallel scans.
#### Effects and tradeoffs
This change preserves source values, supported decoding and SQL
filtering across both readers and read modes. Source fallbacks can fetch
more candidate rows because unsafe predicates remain local. Mixed
aliases containing disabled-source backings can fetch additional source
fields to avoid ES rejecting projected-source requests. Alias scans use
one coordinating scroll to preserve alias semantics; physical-index
parallelism remains. No throughput improvement is claimed. There are no
new public SQL/catalog/session properties or SPI/Thrift/Protobuf
definitions; the optional internal flag uses the existing flat property
map.
#### Known limitation
Mixed-generation FE/BE execution during smooth upgrades is outside this
change's scope. Source plans depending on the updated decoder require
the updated BE for correct results; this PR does not add a
backend-generation gate.
### Release note
Fix Elasticsearch source/index value discrepancies, supported
Boolean/floating/integer source decoding, default epoch-millis date
decoding, disabled-source reads, legacy/configuration-only mapping
handling, retained negation NULL semantics and alias filter/routing
bypass. Preserve eligible doc-value-only queries and reject native-query
combinations requiring local evaluation during planning.
### Check List (For Author)
- Test:
- [x] ES connector reactor package: 163 reported tests, 162 passed, one
optional ES 9 compatibility class skipped; zero failures/errors. All
nine reactor modules passed FE Checkstyle.
- [x] Standard `build.sh --be` ASAN build and all 28
`EsScrollParserTest` unit tests passed. The combined parser/reader run
reported 47 tests: 46 passed and one optional environment-dependent live
smoke test skipped; the separate live regression run exercised both
readers. Clang-format 16, scoped clang-tidy and build hygiene passed for
all affected C++ files. The local BE ELF requires no GLIBC version above
2.10.
- [x] Reproduced the reported alias/script, ARRAY, date-only,
disabled-source, boolean, root-disabled, integer and retained-NOT
failures on the previous head with live ES 8.11.3. FE/BE unit tests also
demonstrated failures before the relevant fixes. Legacy ES6 type
compatibility is covered by mapping-response unit tests.
- [x] Seven live ES 8.11.3 suites passed 1,860 stored checks and 1,764
independent SQL comparisons, preserving 88 expected native planning
errors. All 1,780 prior stored checks remain unchanged.
- [x] `test_es_source_semantics`: 612 stored / 516 independent checks;
`test_es_numeric_source_semantics`: 264 / 264;
`test_es_mapping_boundaries`: 272 / 272;
`test_es_source_fallback_types`: 304 / 304;
`test_es_scalar_materialization`: 160 / 160;
`test_es_source_value_boundaries`: 168 / 168.
- [x] New source-boundary coverage reproduces all six review findings:
JSONB singleton/empty/null-element containers, numeric-double INT64_MIN
and upper endpoint, nullable declared ARRAYs, numeric empty strings,
Java numeric grammar, direct FLOAT rounding, and inherited
malformed/coercion settings with field overrides and mixed aliases. All
eight reader/docvalue/parallel combinations passed. New BE and FE
controls failed before their corresponding fixes.
- [x] `test_es_epoch_millis` adds 80 stored checks and 80 independent
comparisons across both readers, docvalue/parallel modes and UTC/+08:00.
It covers normalized-sibling source fallback, zero/short/fractional
epoch strings, numeric doubles, negative instants, NULL/empty/singleton
arrays, local predicates and pure-docvalue controls. All outputs were
generated by the repository runner and verified without generation; all
80 sections also match an independent date/predicate computation. New
unit coverage checks legacy/V2 date types, declared arrays,
calendar-year precedence and invalid numeric/civil-year bounds.
- [x] New scalar-materialization coverage includes floating/Boolean/date
singleton, empty and null-only arrays; narrow integer rounding
boundaries and NaN, 64-bit precision controls, declared integer ARRAYs
and retained range NOT on multi-value numeric docvalues. New coverage
spans both readers, docvalue modes and parallel modes, source fallback
plus pure-docvalue controls, disabled source indices/mixed aliases,
populated disabled roots, non-UTC date-only values and retained NOT with
optimizer simplification disabled. Session settings are restored. New
outputs were generated by the repository runner and verified separately
without generation; the final date control avoids dependence on the FE
JVM timezone.
- [x] Original FE package built with `build.sh --fe`; connector updates
packaged through the documented module reactor recipe and deployed with
the rebuilt BE for live validation.
- Behavior changed:
- [x] Yes: correctness fallbacks, supported mapping/decoding/query
handling and native-only planning restrictions above.
- Does this need documentation?
- [x] No new public SQL syntax or configuration.
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt