mirror of
https://github.com/apache/doris.git
synced 2026-11-04 07:31:17 +00:00
## Summary Split SQL syntax parsing out of `fe-core` into a new `fe-sql-parser` module that produces an ANTLR parse tree (CST) without semantic analysis. The new module can be packaged as an independent jar for external consumers (third-party tools, linters, format converters, etc.) without dragging in `LogicalPlan`, `Catalog`, `ConnectContext`, or any other fe-core internals. ### Module changes - Move `DorisLexer.g4` / `DorisParser.g4` and 8 supporting Java files (`CaseInsensitiveStream`, `Origin`, `ParserUtils`, `ParseErrorListener`, `PostProcessor`, `ParseException`, `SyntaxParseException`, `QueryParsingErrors`) into `fe-sql-parser`. Package names are preserved so fe-core's hundreds of imports do not move. - `ParseException` now extends `RuntimeException` directly to break the chain through `nereids.exceptions.AnalysisException`, which references `LogicalPlan`. - Introduce an `OriginAware` SPI in `fe-sql-parser` so `ParserUtils` keeps its per-thread field-based fast path (originally added in #52125). `MoreFieldsThread` in fe-core implements the interface; threads that don't fall back to ThreadLocal — correctness is identical either way. - Add `org.apache.doris.sqlparser.DorisSqlParser` facade with `parseStatement` / `parseStatements` / `parseExpression`. - fe-core reverse-depends on fe-sql-parser; its own `antlr4-maven-plugin` now only processes the Nereids pattern-generator's `JavaLexer.g4` / `JavaParser.g4`. The new module's only runtime dependency is `org.antlr:antlr4-runtime`. ### Standalone CLI `mvn -pl fe-sql-parser -Pcli package` produces a self-contained executable jar (`fe-sql-parser-*-cli.jar`, ~1.7 MB after minimize-shade) so the parser can be invoked directly from a shell: ```bash $ java -jar fe-sql-parser-*-cli.jar "SELECT a FROM t WHERE a > 1" (singleStatement (statement ...) <EOF>) $ java -jar fe-sql-parser-*-cli.jar --pretty --multi "USE db; SELECT 1; SELECT 2" $ java -jar fe-sql-parser-*-cli.jar --expression "a + 1 * COALESCE(b, 0)" $ echo "SELECT 1" | java -jar fe-sql-parser-*-cli.jar $ java -jar fe-sql-parser-*-cli.jar -f query.sql ``` The CLI is gated behind the `cli` Maven profile so default Doris builds do not pay the shading cost; the thin jar consumed by fe-core is unchanged. Exit codes: `0` success, `1` parse error, `2` usage or I/O error. ### Extension hooks for downstream tools Downstream projects can plug in custom logic (SQL lineage, policy enforcement, audit, rewriting, metrics) **without modifying `fe-sql-parser`**. Four mechanisms are available: | Mechanism | When it fires | Typical use | |-----------|---------------|-------------| | Subclass `DorisParserBaseVisitor<T>` | After parsing | Extract information, rewrite, lineage | | Subclass `DorisParserBaseListener` | After parsing | Simple `enter`/`exit` interception | | `parser.addParseListener(...)` via `newLexer` / `newParser` | Live, during parsing | Token-level processing, on-the-fly mutation | | Wrap `DorisSqlParser` | Around the call | Metrics, caching, request-level policy | `fe/fe-sql-parser/README.md` contains end-to-end examples for SQL lineage extraction, policy/audit listeners, hint collection during parsing, an instrumented facade with caching and metrics, plus tips on locating rule names and debugging visitors with the CLI. It also documents the build modes, library/Visitor usage, configuration flags (`noBackslashEscapes`, `ansiSqlSyntax`), the `OriginAware` fast-path SPI, and current caveats. ## Test plan - [x] `fe-sql-parser` unit tests: 7 new cases in `DorisSqlParserTest` covering `SELECT`, `SELECT FROM WHERE`, multi-statement, expression, DDL (`CREATE TABLE` with `DISTRIBUTED` + `PROPERTIES`), malformed SQL, and trailing-garbage expressions - [x] Full `fe` reactor compiles (`mvn -pl fe-core -am compile`) - [x] fe-core's 15 existing `org.apache.doris.nereids.parser.*Test` classes pass (160 cases, 0 failures) - [x] CLI smoke test: positional / `-e` / `-f` / stdin input modes; `--multi` / `--expression` parse modes; `--pretty` output; parse-error exit code --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
166 lines
7.1 KiB
XML
166 lines
7.1 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!--
|
|
Licensed to the Apache Software Foundation (ASF) under one
|
|
or more contributor license agreements. See the NOTICE file
|
|
distributed with this work for additional information
|
|
regarding copyright ownership. The ASF licenses this file
|
|
to you under the Apache License, Version 2.0 (the
|
|
"License"); you may not use this file except in compliance
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing,
|
|
software distributed under the License is distributed on an
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations
|
|
under the License.
|
|
-->
|
|
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
|
|
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
|
<modelVersion>4.0.0</modelVersion>
|
|
<parent>
|
|
<groupId>org.apache.doris</groupId>
|
|
<version>${revision}</version>
|
|
<artifactId>fe</artifactId>
|
|
<relativePath>../pom.xml</relativePath>
|
|
</parent>
|
|
<artifactId>fe-sql-parser</artifactId>
|
|
<packaging>jar</packaging>
|
|
<name>Doris FE SQL Parser</name>
|
|
<description>Standalone ANTLR4-based SQL syntax parser for Apache Doris. Produces a parse tree (CST) without
|
|
any semantic analysis. Designed to be consumed as an independent jar by external tools.</description>
|
|
|
|
<dependencies>
|
|
<dependency>
|
|
<groupId>org.antlr</groupId>
|
|
<artifactId>antlr4-runtime</artifactId>
|
|
<version>${antlr4.version}</version>
|
|
</dependency>
|
|
<!-- Test dependencies -->
|
|
<dependency>
|
|
<groupId>org.junit.jupiter</groupId>
|
|
<artifactId>junit-jupiter</artifactId>
|
|
<scope>test</scope>
|
|
</dependency>
|
|
</dependencies>
|
|
|
|
<build>
|
|
<finalName>doris-fe-sql-parser</finalName>
|
|
<directory>${project.basedir}/target/</directory>
|
|
<plugins>
|
|
<!-- antlr -->
|
|
<plugin>
|
|
<groupId>org.antlr</groupId>
|
|
<artifactId>antlr4-maven-plugin</artifactId>
|
|
<version>${antlr4.version}</version>
|
|
<executions>
|
|
<execution>
|
|
<id>antlr</id>
|
|
<goals>
|
|
<goal>antlr4</goal>
|
|
</goals>
|
|
</execution>
|
|
</executions>
|
|
<configuration>
|
|
<visitor>true</visitor>
|
|
<sourceDirectory>src/main/antlr4</sourceDirectory>
|
|
<treatWarningsAsErrors>true</treatWarningsAsErrors>
|
|
</configuration>
|
|
</plugin>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-jar-plugin</artifactId>
|
|
<executions>
|
|
<execution>
|
|
<id>prepare-test-jar</id>
|
|
<phase>test-compile</phase>
|
|
<goals>
|
|
<goal>test-jar</goal>
|
|
</goals>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-javadoc-plugin</artifactId>
|
|
<configuration>
|
|
<skip>true</skip>
|
|
</configuration>
|
|
</plugin>
|
|
</plugins>
|
|
</build>
|
|
<profiles>
|
|
<profile>
|
|
<!-- Build a self-contained executable jar with antlr4-runtime bundled and a
|
|
Main-Class manifest. Produces target/doris-fe-sql-parser-cli.jar.
|
|
Usage: java -jar target/doris-fe-sql-parser-cli.jar "SELECT 1" -->
|
|
<id>cli</id>
|
|
<build>
|
|
<plugins>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-shade-plugin</artifactId>
|
|
<executions>
|
|
<execution>
|
|
<id>build-cli-jar</id>
|
|
<phase>package</phase>
|
|
<goals>
|
|
<goal>shade</goal>
|
|
</goals>
|
|
<configuration>
|
|
<shadedArtifactAttached>true</shadedArtifactAttached>
|
|
<shadedClassifierName>cli</shadedClassifierName>
|
|
<createDependencyReducedPom>false</createDependencyReducedPom>
|
|
<minimizeJar>true</minimizeJar>
|
|
<transformers>
|
|
<transformer
|
|
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
|
|
<mainClass>org.apache.doris.sqlparser.DorisSqlParserCli</mainClass>
|
|
</transformer>
|
|
</transformers>
|
|
<filters>
|
|
<filter>
|
|
<artifact>*:*</artifact>
|
|
<excludes>
|
|
<exclude>META-INF/*.SF</exclude>
|
|
<exclude>META-INF/*.DSA</exclude>
|
|
<exclude>META-INF/*.RSA</exclude>
|
|
<exclude>module-info.class</exclude>
|
|
</excludes>
|
|
</filter>
|
|
</filters>
|
|
</configuration>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
</plugins>
|
|
</build>
|
|
</profile>
|
|
<profile>
|
|
<id>release</id>
|
|
<build>
|
|
<plugins>
|
|
<plugin>
|
|
<groupId>org.apache.maven.plugins</groupId>
|
|
<artifactId>maven-source-plugin</artifactId>
|
|
<configuration>
|
|
<attach>true</attach>
|
|
</configuration>
|
|
<executions>
|
|
<execution>
|
|
<id>create-source-jar</id>
|
|
<goals>
|
|
<goal>jar-no-fork</goal>
|
|
<goal>test-jar-no-fork</goal>
|
|
</goals>
|
|
</execution>
|
|
</executions>
|
|
</plugin>
|
|
</plugins>
|
|
</build>
|
|
</profile>
|
|
</profiles>
|
|
</project>
|