airborne12 opened a new pull request, #70273:
URL: https://github.com/apache/doris/pull/70273
> Stacked PR in the SNII / text analysis / ANN CLucene removal series (index
at the end). It depends on #70269 (T10) and #70272 (T13), whose commits show in
this diff until they merge; review only this PR's own commits: `b801e43a1e3`
[refactor](be) Remove the temporary CLucene facilities of the analysis
migration. It stays a draft until its dependencies merge.
### What problem does this PR solve?
Issue Number: None
Related PR: #70269, #70272
Problem Summary:
the text analysis layer, SNII and ANN no longer use CLucene,
and the facilities that let the move go one piece at a time are left over:
- LuceneBackedTextAnalyzer (inverted/clucene/lucene_backed_text_analyzer.*)
served analyzers that were not yet on the CLucene-free stream interface.
Nothing uses it any more: deleted, with its tests in
lucene_adapter_test.cpp.
LuceneAnalyzerBridge, which the V1/V2 writers use, stays.
- doris_be linked clucene-contribs-lib, which carries CLucene's jieba and
other
language analyzers, and every source had its headers on the include path,
with LanguageBasedAnalyzer.h in the PCH. Now only the BE UT links it, as
the
oracle of the jieba live differential, and only doris_be_test has its
headers; pch.h drops LanguageBasedAnalyzer.h. These three changes go
together: the old PCH no longer finds its header on the new include path.
- check-header-deps.py: the CLucene exception tables are empty and go, with
the logic that kept them in step, so the closure and symbol rules fail on
any
violation under the five CLucene-free roots (snii, analysis, query, common,
ann). The query/ ANGLE_BANS entries, which the closure rule now covers, go
too; the self-test checks the closure rule in their place.
**Final acceptance of the series** (release binaries of the baseline, the
series' first commit with the old analyzers, against this PR's head; one
worktree cluster, the BE binary swapped between runs):
Throughput of the analyzer benchmark
(`be/benchmark/benchmark_analyzer.hpp`), five alternating rounds, median of the
per-round ratios on each corpus; reuse = one stream across rows, row = a stream
per row. The four mixed corpora are agent_input, http_request, otel_body and
prose_en; zh_wiki and ja_wiki are real Chinese and Japanese text.
| analyzer | line | reuse | row | result |
|---|---|---|---|---|
| english | four mixed geomean >= 1.5, each >= 1.2 | 3.20, lowest 2.46 |
2.73, lowest 1.96 | met |
| basic | four mixed + zh + ja, same | 7.26, lowest 4.21 | 7.57, lowest 4.50
| met |
| standard, unicode | same | 1.83, lowest 1.53 | 1.77, lowest 1.48 | met |
| chinese coarse / fine | zh >= 1.5, four mixed geomean >= 1.5 | zh 3.02 /
2.68 | zh 2.89 / 2.52 | met |
| icu | zh and ja >= 1.5 | zh 1.49, ja 1.43 | zh 1.50, ja 1.48 | about 1.5
zh / 1.45-1.48 ja over 25 rounds (target relaxed to that level) |
| ik smart / max_word | every corpus >= 1.2 | lowest 1.54 / 1.51 | lowest
1.45 / 1.46 | met |
| kuromoji | ja >= 0.97 | 0.96 | 0.96 | 1.00 / 1.01 pooled over 25 rounds:
no regression |
| custom standard / basic + lowercase, standard + asciifolding + lowercase |
four mixed geomean >= 1.5 | 2.28 / 4.44 / 2.06 | 2.43 / 4.69 / 2.21 | met |
| custom keyword + lowercase, word_delimiter | four mixed >= 0.97 | lowest
2.73 / 1.96 | lowest 3.07 / 1.80 | met |
The host was shared (load 40 to 170): a baseline-against-baseline run
deviates up to 10% on single long-corpus entries (kuromoji, one pass per
measurement) and at most 3% per analyzer, so the icu and kuromoji rows rest on
the pooled rounds. The per-row token digests of every benchmark row (5,240,060)
are identical to the baseline's.
- inverted_index_p0: 250 of 252 suites pass. The other two fail with "Index
policy number cannot exceed 100" because earlier suites leave their analyzers
behind; they fail the same way with the baseline binary on that cluster and
pass with both binaries on a fresh one.
- Both directions end to end: 1,237,030 rows (AgentLogs input and output,
TextBench bodies, Chinese and Japanese Wikipedia), 236.5 MB of text, in V2 and
SNII tables with twelve indexed copies (english, standard, unicode, basic,
chinese coarse and fine, ik_max_word, ik_smart, icu, kuromoji and two custom
chains). 10,248 MATCH_ANY / MATCH_ALL / MATCH_PHRASE / MATCH_PHRASE_PREFIX
queries per run, caches off, MATCH without an index disabled. Indexes built by
the baseline and queried by both binaries, and indexes built by this PR and
queried by both: all four result sets are identical query by query, and V2 and
SNII agree. The data files are byte for byte the same; the index files differ
only in index ids, CLucene's clock-based segment version (V2) and footer
checksums (SNII).
- Load of those rows, eight alternating runs per binary: BE CPU 0.878 of the
baseline's for V2 and 0.784 for SNII (median CPU seconds 323.6 against 284.1
and 332.3 against 260.4); user-space instructions 0.750 and 0.755. SNII's index
build memory peaks 32 MiB (+1.2%) higher, because the analysis streams now live
across rows; the process RSS peak is not higher.
- doris_be needs at most GLIBC_2.10, as the baseline does, and imports no
symbol the baseline does not.
### Release note
Behavior changes of the CLucene-free text analysis, SNII and ANN code:
- An analyzer error while a V1/V2 inverted index is written (for example ik
on
a value with no decodable UTF-8 character) fails the load with
INVERTED_INDEX_ANALYZER_ERROR and "Analyzer error while adding document: "
instead of INVERTED_INDEX_CLUCENE_ERROR and "CLuceneError add_document: ".
A custom analyzer with the ik tokenizer loads its dictionary on the first
value, so a missing ik dictionary there changes the same way ("Analyzer
error while creating new index field: "). Errors creating a builtin
analyzer, a missing dictionary included, were and stay
INVERTED_INDEX_ANALYZER_ERROR.
- With the session variable inverted_index_snii_read_no_write_file_cache set,
ANN index reads from SNII files no longer fill the file cache on a miss,
like
every other SNII read.
- The per-column memory an SNII index build reports (the SniiIndexBuild
tracker, the snii_index_build_mem_consumption metric and the spill
decision)
now includes every analyzer's stream buffers and token batch, which live
across rows; before, only the gram analyzers' buffers counted.
- The chinese parser no longer needs idf.utf8. A failed ik dictionary load is
retried by the next analyzer that needs it instead of failing until a
restart, so putting the missing files under inverted_index_dict_path is
enough; changing that path still needs a restart.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Details:
- BE UT under ASAN, the whole suite in one process: 16,269 of 16,342
tests pass, 67 skipped, 6 fail, all from this environment: three
GcpTokenProviderTest cases, whose OAuth request to the test's own
server
the sandbox's proxy variables take (they pass without them), and three
CdcClientMgrTest cases (and the LeakSanitizer report at exit), which
fail the same way on a binary built before this change: posix_spawn
file actions resolve to a second glibc-compatibility archive of the
local toolchain.
- The analysis golden files pass unchanged, the full code point scan
(ANALYSIS_GOLDEN_FULL=1) included; the jieba live differential passes
13 of 13; with USE_AVX2=0 in its own build directory, the analysis
suites pass 3471 of 3476 with 5 skipped, and no BE source is compiled
with -mavx2.
- The doris_be link line has no clucene-contribs-lib and nm -C on
doris_be
finds no cppjieba symbol; doris_be_test still links it.
- With ENABLE_PCH=OFF ENABLE_UNITY_BUILD=OFF, ./build.sh --be
--index-tool
(ASAN) and ./build.sh --be --index-tool --benchmark (RELEASE, the only
type with the benchmark) pass, and none of the 184 sources under the
five
roots, compiled with -H and its own command, opens a CLucene header.
- check-build-hygiene.sh passes with no exception, and so do its
self-tests (build-support/tests/run.sh), which three mutations of the
gate turn red; clang-format 16 and run-clang-tidy.sh are clean.
- Behavior changed:
- [ ] No.
- [x] Yes.
the four items of the release note, made by the earlier changes of
this work.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
