Jack
2524893ea3
### What problem does this PR solve? Issue Number: N/A Related PR: #69399 (merged; the plan copier fix this PR needs), #69419 (merged; fixes the pre-existing MATCH predicate inference that review of this PR found) Problem Summary: `score()` only worked as `ORDER BY score() LIMIT n` directly above the scan whose MATCH defines it. A minimum score filter without that TopN evaluated the unbound `score()` placeholder as zero and silently dropped every row: ```sql SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001; -- returned nothing ``` `score()` is computed by the olap scan that evaluates the MATCH and SEARCH predicates of its SELECT. This PR binds `score()` to that scan at the start of rewrite, makes the scan's expressions read the score, and rejects every other use with an error that names the rule. **Supported.** A SELECT that reads one table and has a MATCH or SEARCH predicate in its WHERE clause can use `score()`: - in the WHERE clause, as one minimum score: `score() > x` or `score() >= x` (also `x < score()`, `x <= score()`, and with a FLOAT or DOUBLE cast); - in the select list, window functions and ORDER BY, if the SELECT does not aggregate (with GROUP BY, including GROUPING SETS, ROLLUP and CUBE, aggregate functions or DISTINCT, `score()` can only be the minimum score). A scored SELECT in a subquery, view, CTE or set operation still needs `ORDER BY score() LIMIT n`, as before. The TopN keeps the predicates of the outer query out of its scan, e.g. an outer MATCH cannot add to its score. A correlated subquery cannot call `score()`, because decorrelation removes that TopN. **Rejected.** A join, a subquery or a lateral view in the scored SELECT; a view, CTE or other relation that is not an olap table (a table stream fails during planning, because its scan is replaced by scans of the table and of its changes); any other filter on the score (`score() < x`, `score() = x`, lossy casts, ORs, a second bound); `score()` in aggregates, GROUP BY, HAVING or QUALIFY; MATCH or SEARCH in HAVING or QUALIFY of a scored SELECT, also through an alias; `score()` in a correlated subquery; a SELECT without its own MATCH or SEARCH; a nested scored SELECT without the TopN. **How.** - BE: the scan reads the score in its own expressions. `SegmentIterator` prepares the score of every row of its row bitmap before the first batch. It marked only stored columns of a pushed-down expression as common-expression columns, so an expression on the score alone, e.g. `score() > 0.5`, was never evaluated (and the scanner had already removed it from its conjuncts), and an expression that read the score next to other columns, e.g. a TopN filter on `CAST(score() AS INT)`, evaluated `Score()` itself, which returns 0. Now the score column is read with the other common-expression columns, before the expressions run. - FE, binding: `BindScoreToOlapScan` replaces `CheckScoreUsage`. It runs first in rewrite, while subqueries, views and CTEs still have their own plan nodes, so the analysis classes are as on master. For each SELECT that calls `score()`, it checks the rules above, adds a score column to the scan (score limit 0: the scan scores every row it returns) and replaces the `score()` calls of the SELECT with that column. A table is a relation, also under the filters analysis puts on it (the delete sign filter of a UNIQUE table, row policies). Binding after analysis also covers `* REPLACE (score() AS c)`, alias functions over `score()` and `ORDER BY` a position, and a UDF named `score` is never mistaken for it. - FE, rewrite: the minimum score is an ordinary predicate of the scan. `PushDownScoreTopNIntoOlapScan` pushes a TopN on the score into the scan, and with it the minimum score, exactly as before this PR (with a zero score limit when SEARCH has another predicate), because the scan keeps the best rows of a segment only when no predicate follows. TopN filter pushdown and MV rewrite need no `score()` cases: a TopN filter reads the score like any other column, and a materialized view without the score cannot answer a predicate on it. - This PR needs the plan copier fix of #69399, which is merged: copying a scan with virtual columns gave each one two ids and kept the old slot in the output. - The BE keeps the score unchanged by MATCH projections: `SegmentIterator` evaluates them without the score collector. - `SearchExpression` can be rendered as SQL, which the shape plan logged at debug level and `EXPLAIN SHAPE PLAN` need. ### Release note `score()` works in the select list, window functions and ORDER BY of a SELECT on one table whose WHERE clause has MATCH or SEARCH, with an optional minimum score `score() > x` or `score() >= x`, without `ORDER BY score() LIMIT`. A scored SELECT in a subquery, view, CTE or set operation still needs `ORDER BY score() LIMIT`. Other uses of `score()` are rejected instead of evaluating it as 0.
🌍 Read this in other language
English • العربية • বাংলা • Deutsch • Español • فارسی • Français • हिन्दी • Bahasa Indonesia • Italiano • 日本語 • 한국어 • Polski • Português • Română • Русский • Slovenščina • ไทย • Türkçe • Українська • Tiếng Việt • 简体中文 • 繁體中文
?style=for-the-badge){kind=icon}
?style=for-the-badge){kind=icon}
Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It provides fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.
Explore the official website for the latest product overview, use cases, ecosystem updates, blogs, and user stories. For version updates, see all release notes.
📈 Use Cases
| Use Case | What it provides |
|---|---|
| Customer-Facing Analytics | Ship sub-second interactive analytics to external users. |
| Data Warehousing | Build one real-time warehouse across business domains. |
| Observability | Analyze high-throughput logs, events, and metrics with SQL. |
| Doris for AI | Use vector, text, JSON, and structured search in one SQL engine. |
🚀 Core Capabilities
Apache Doris is built around three core capabilities. The website is the source of truth for detailed product descriptions and examples.
| Capability | What it provides |
|---|---|
| Real-Time Analytics | Streaming ingestion, incremental transformation, and sub-second queries under high concurrency. |
| Lakehouse Analytics | Fast SQL analytics over open table formats such as Iceberg, Delta Lake, and Hudi. |
| Hybrid Search | SQL-native analytics across JSON, full-text, and vector data for AI and search workloads. |
🔌 Ecosystem
Doris sits at the center of the modern data stack. It connects upstream databases, streaming systems, and lakehouse storage with downstream BI, AI, analytics, and observability tools.
For the latest ecosystem coverage, visit the official website and the connection and integration documentation.
👣 Get Started
🧱 Architecture
Apache Doris supports both compute-storage coupled and compute-storage decoupled deployments. In decoupled mode, stateless compute groups run over shared object storage, so you can scale compute on demand and isolate workloads.
Learn more in the deployment guide and deployment mode guide.
📣 Project Updates
| Resource | What it provides |
|---|---|
| Community Report | Weekly updates on community activity, merged PRs, contributors, and feature progress. |
| Roadmap 2026 | The 2026 planning discussion for AI and hybrid search, query engine, storage, and data lake work. |
🧩 Components
Doris provides connectors and tools for common data engineering workflows.
- Doris Flink Connector
- Doris Spark Connector
- Doris Kafka Connector
- Doris Stream Loader
- Doris Kubernetes Operator
👨👩👧👦 Users
Apache Doris is used in production by thousands of companies worldwide across internet services, finance, retail, logistics, manufacturing, energy, telecommunications, AI, and other industries.
🙌 Contributors
Apache Doris graduated from the Apache Incubator and became an Apache Top-Level Project in June 2022. Thanks to all community contributors who help build Doris.
🌈 Community and Support
💬 Contact Us
🧰 Links
- Apache Doris official website: doris.apache.org
- X: @doris_apache
- Medium: @ApacheDoris
📜 License
Note
Some licenses of the third-party dependencies are not compatible with Apache 2.0 License. So you need to disable some Doris features to comply with Apache 2.0 License. For details, refer to the
thirdparty/LICENSE.txt