[fix](inverted index) update clucene to reject terms containing an embedded NUL byte (#67179)

Browse Source

Bumps contrib/clucene from abe2b71a1c1 to 08d45e541fe, picking up
apache/doris-thirdparty#368.

Problem
-------
BE dies with SIGSEGV while writing an inverted index when a term
contains a
raw NUL byte in the middle (JSON \u0000 decodes to this). The crash
reproduces
on any table with an inverted index over such data, and restarts in a
loop.

Root cause
----------
In SDocumentWriter.cpp the two paths disagree on what a term is:

* addPosition() hashes and dedups with eq(p->term_, term), i.e. the FULL
string_view length, so "svc\0A" and "svc\0B" are two distinct postings;
  * comparePostings() stops at CLUCENE_END_OF_WORD, so the very same two
    postings compare EQUAL.

A run of mutually "equal" but distinct postings makes quickSort()'s
partition
loop peel exactly one element per call, so the recursion depth becomes
O(n)
(measured depth/N = 1.000). Each frame is 128 bytes, so ~65k levels
exhaust the
8 MiB thread stack. The stack is so far gone that the kernel cannot
build a
signal frame and force-kills the process (si_code=SI_KERNEL), which is
why no
handler output or stack trace ever reaches the log.

Fix
---
doris-thirdparty#368 rejects such terms in addPosition(), before they
can enter
the postings hash, so the degenerate equal-run can no longer form. The
row now
fails with a clear CLuceneError, which
InvertedIndexColumnWriter::add_document()
already converts into INVERTED_INDEX_CLUCENE_ERROR, instead of taking
the BE down.

Verification
------------
Reproducer: 80k terms shaped "svc\0<i>", untokenized inverted index, 8
MiB stack.

  before: SIGSEGV, exit 139
  after : 80000 rows rejected, exit 0, 0 postings reach the sort

Performance on clean data (the guard is one compare + a never-taken
branch):

  perf instructions, 1M tokens : 10,116,997,256 -> 10,118,997,132
                                 = +2.00 instructions/token (+0.020%)
  index output (.fnm/.frq/.nrm/.tii/.tis) : byte-identical

...

This commit is contained in:

Jack authored and GitHub committed 2026-08-27 10:01:20 +08:00

1 parent 4d7a207363

commit e59320b957

1 file changed

+1 -1

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

contrib/clucene

+1 -1

Submodule contrib/clucene updated: abe2b71a1c...08d45e541f.