Files
Mingyu Chen (Rayner)andClaude Opus 4.7 7fc8e27628 [feat](sql-parser) Split SQL grammar into standalone fe-sql-parser (#63823)
## Summary

Split SQL syntax parsing out of `fe-core` into a new `fe-sql-parser`
module that produces an ANTLR parse tree (CST) without semantic
analysis. The new module can be packaged as an independent jar for
external consumers (third-party tools, linters, format converters, etc.)
without dragging in `LogicalPlan`, `Catalog`, `ConnectContext`, or any
other fe-core internals.

### Module changes

- Move `DorisLexer.g4` / `DorisParser.g4` and 8 supporting Java files
(`CaseInsensitiveStream`, `Origin`, `ParserUtils`, `ParseErrorListener`,
`PostProcessor`, `ParseException`, `SyntaxParseException`,
`QueryParsingErrors`) into `fe-sql-parser`. Package names are preserved
so fe-core's hundreds of imports do not move.
- `ParseException` now extends `RuntimeException` directly to break the
chain through `nereids.exceptions.AnalysisException`, which references
`LogicalPlan`.
- Introduce an `OriginAware` SPI in `fe-sql-parser` so `ParserUtils`
keeps its per-thread field-based fast path (originally added in #52125).
`MoreFieldsThread` in fe-core implements the interface; threads that
don't fall back to ThreadLocal — correctness is identical either way.
- Add `org.apache.doris.sqlparser.DorisSqlParser` facade with
`parseStatement` / `parseStatements` / `parseExpression`.
- fe-core reverse-depends on fe-sql-parser; its own
`antlr4-maven-plugin` now only processes the Nereids pattern-generator's
`JavaLexer.g4` / `JavaParser.g4`.

The new module's only runtime dependency is `org.antlr:antlr4-runtime`.

### Standalone CLI

`mvn -pl fe-sql-parser -Pcli package` produces a self-contained
executable jar (`fe-sql-parser-*-cli.jar`, ~1.7 MB after minimize-shade)
so the parser can be invoked directly from a shell:

```bash
$ java -jar fe-sql-parser-*-cli.jar "SELECT a FROM t WHERE a > 1"
(singleStatement (statement ...) <EOF>)

$ java -jar fe-sql-parser-*-cli.jar --pretty --multi "USE db; SELECT 1; SELECT 2"
$ java -jar fe-sql-parser-*-cli.jar --expression "a + 1 * COALESCE(b, 0)"
$ echo "SELECT 1" | java -jar fe-sql-parser-*-cli.jar
$ java -jar fe-sql-parser-*-cli.jar -f query.sql
```

The CLI is gated behind the `cli` Maven profile so default Doris builds
do not pay the shading cost; the thin jar consumed by fe-core is
unchanged. Exit codes: `0` success, `1` parse error, `2` usage or I/O
error.

### Extension hooks for downstream tools

Downstream projects can plug in custom logic (SQL lineage, policy
enforcement, audit, rewriting, metrics) **without modifying
`fe-sql-parser`**. Four mechanisms are available:

| Mechanism | When it fires | Typical use |
|-----------|---------------|-------------|
| Subclass `DorisParserBaseVisitor<T>` | After parsing | Extract
information, rewrite, lineage |
| Subclass `DorisParserBaseListener` | After parsing | Simple
`enter`/`exit` interception |
| `parser.addParseListener(...)` via `newLexer` / `newParser` | Live,
during parsing | Token-level processing, on-the-fly mutation |
| Wrap `DorisSqlParser` | Around the call | Metrics, caching,
request-level policy |

`fe/fe-sql-parser/README.md` contains end-to-end examples for SQL
lineage extraction, policy/audit listeners, hint collection during
parsing, an instrumented facade with caching and metrics, plus tips on
locating rule names and debugging visitors with the CLI. It also
documents the build modes, library/Visitor usage, configuration flags
(`noBackslashEscapes`, `ansiSqlSyntax`), the `OriginAware` fast-path
SPI, and current caveats.

## Test plan

- [x] `fe-sql-parser` unit tests: 7 new cases in `DorisSqlParserTest`
covering `SELECT`, `SELECT FROM WHERE`, multi-statement, expression, DDL
(`CREATE TABLE` with `DISTRIBUTED` + `PROPERTIES`), malformed SQL, and
trailing-garbage expressions
- [x] Full `fe` reactor compiles (`mvn -pl fe-core -am compile`)
- [x] fe-core's 15 existing `org.apache.doris.nereids.parser.*Test`
classes pass (160 cases, 0 failures)
- [x] CLI smoke test: positional / `-e` / `-f` / stdin input modes;
`--multi` / `--expression` parse modes; `--pretty` output; parse-error
exit code

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-29 14:30:16 +08:00

166 lines
7.1 KiB
XML

<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.doris</groupId>
<version>${revision}</version>
<artifactId>fe</artifactId>
<relativePath>../pom.xml</relativePath>
</parent>
<artifactId>fe-sql-parser</artifactId>
<packaging>jar</packaging>
<name>Doris FE SQL Parser</name>
<description>Standalone ANTLR4-based SQL syntax parser for Apache Doris. Produces a parse tree (CST) without
any semantic analysis. Designed to be consumed as an independent jar by external tools.</description>
<dependencies>
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr4-runtime</artifactId>
<version>${antlr4.version}</version>
</dependency>
<!-- Test dependencies -->
<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<finalName>doris-fe-sql-parser</finalName>
<directory>${project.basedir}/target/</directory>
<plugins>
<!-- antlr -->
<plugin>
<groupId>org.antlr</groupId>
<artifactId>antlr4-maven-plugin</artifactId>
<version>${antlr4.version}</version>
<executions>
<execution>
<id>antlr</id>
<goals>
<goal>antlr4</goal>
</goals>
</execution>
</executions>
<configuration>
<visitor>true</visitor>
<sourceDirectory>src/main/antlr4</sourceDirectory>
<treatWarningsAsErrors>true</treatWarningsAsErrors>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<executions>
<execution>
<id>prepare-test-jar</id>
<phase>test-compile</phase>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<!-- Build a self-contained executable jar with antlr4-runtime bundled and a
Main-Class manifest. Produces target/doris-fe-sql-parser-cli.jar.
Usage: java -jar target/doris-fe-sql-parser-cli.jar "SELECT 1" -->
<id>cli</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<executions>
<execution>
<id>build-cli-jar</id>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<shadedClassifierName>cli</shadedClassifierName>
<createDependencyReducedPom>false</createDependencyReducedPom>
<minimizeJar>true</minimizeJar>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.apache.doris.sqlparser.DorisSqlParserCli</mainClass>
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
<exclude>module-info.class</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<id>release</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<configuration>
<attach>true</attach>
</configuration>
<executions>
<execution>
<id>create-source-jar</id>
<goals>
<goal>jar-no-fork</goal>
<goal>test-jar-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>