airborne12 opened a new pull request, #70269:
URL: https://github.com/apache/doris/pull/70269
> Stacked PR in the SNII / text analysis / ANN CLucene removal series (index
at the end). It depends on #70268 (T9), whose commits show in this diff until
they merge; review only this PR's own commits: `622873012b7` [improvement](be)
Rewrite the tokens of a batch in place in the one-to-one filters. It stays a
draft until its dependencies merge.
### What problem does this PR solve?
Issue Number: None
Related PR: #70268
Problem Summary:
lowercase, icu_normalizer and asciifolding were per-token
filters: each pulled the tokens of its input one at a time into a batch of
its
own and copied every term it handed on, changed or not. They now rewrite the
tokens of their input's batch where they stand, writing only a changed term
into the batch's memory. word_delimiter, pinyin and asciifolding with
preserve_original, which can make two tokens out of one, stay per token. The
tokens and the exceptions do not change.
- lower_case_filter.h: LowerCaseFilter is a TextTokenFilter with a native
next_batch(). Which terms it rewrites decides where it throws, so a term is
left as it is only when it is certainly valid: ASCII without a capital, or
exactly one 3-byte character with a lead byte of 0xE3 to 0xE9 and two
continuation bytes, which lies in U+3000 to U+9FFF and has no case. An
ASCII
term with a capital is lowercased byte by byte into the batch; any other
term
takes the old path, validation (the same exception) and ICU's root-locale
case mapping into the batch, retried once at the length ICU asks for when
the
term grows (as İ does).
- icu_normalizer_filter: native and one to one; a term the quick check passes
keeps its view, any other is normalized as before and copied into the
batch;
a failed normalization still logs and leaves the token as it is.
- ascii_folding_filter: ASCIIFoldingFilter is the native one-to-one filter,
keeping the view of a term with nothing to fold; the per-token code stays
as
ASCIIFoldingPreserveOriginalFilter, which the factory makes for
preserve_original=true; both share may_fold() and fold().
The filters hand out the batches their input hands out instead of refilling a
batch of 256 tokens, so an exception now comes out of the call that reaches
the
bad token, after the batches before it, where a filter used to throw before
handing out the tokens ahead of it. The same values throw, with the same code
and message.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Details:
- BE UT under ASAN: the analysis, index and search suites with the T5 to
T10 ones, 4222 of 4229 tests pass and the 7 skipped are the opt-in
ones;
the code point scan passes with ANALYSIS_GOLDEN_FULL=1. New:
OneToOneFilterTest (each batch filter against a per-token oracle of the
replaced code on random streams of 1 to 4 batches of 1 to 300 tokens,
with all four classes, empty terms, invalid UTF-8 and terms held as
views
or as batch copies: the same terms, increments, codes and messages; for
icu_normalizer in four settings; terms left as they are keep their
views), LowerCaseFilterTest.HandlesEachClassOfTerm (the four classes,
which terms keep their views, and that only five go through ICU),
LowerCaseFilterTest.RejectsALeadWithoutContinuationBytes (keyword +
lowercase on E4 41 41 and four other malformed terms throws
INVERTED_INDEX_ANALYZER_ERROR) and, in ASCIIFoldingFilterFactoryTest,
standard + asciifolding on "café" with preserve_original ("cafe", then
"café" with an increment of 0) and without it.
RejectsInvalidUtf8WithAnalyzerError now reads the valid batch before
the
call that throws.
- The analysis golden files are unchanged, and the benchmark's digest
mode
gives the same tokens as the T1 baseline binary in all 5,240,060 rows,
the custom chains with lowercase and asciifolding among them.
- The new filters were also compared with the old ones directly, outside
the BE build, behind the real standard, basic, keyword, icu and empty
tokenizers in 32 chains (the benchmark's custom chains, each tokenizer
before each filter, every icu_normalizer setting, both asciifolding
settings, filters in a row): all corpora, every BMP code point in 15
contexts, supplementary code points, every 3-byte sequence led by 0xE0
to 0xEF and 60k random values with invalid bytes per chain. 137.1M
values, 773M tokens and 923K values that threw: 0 differences in tokens
and exceptions; 57.1M values under ASAN/UBSan, 0 differences and no
sanitizer report. 10 mutants of the new code were all killed, and no
character in U+3000 to U+9FFF has a lowercase mapping (checked for each
one).
- Throughput of the benchmark binary of this change with
be/benchmark/analyzer_ab.py: five alternating rounds, the median of the
per-round ratios on each corpus and their geometric mean, reuse/ and
row/, at a host load of 37 to 206, so the ranges are wide. Against the
T1
baseline: custom standard+lowercase 2.97 and 3.11, basic+lowercase 4.68
and 4.94, standard+asciifolding+lowercase 2.49 and 2.62 (the three
together 3.26 and 3.43); keyword+lowercase 2.31 and 2.57 (ja_wiki 1.09
and 1.23, zh_wiki 1.12 and 1.27, where ICU's case mapping dominates
long
values), standard+word_delimiter+lowercase 2.62 and 2.45. Against T9:
the
three 1.67 to 2.04 and 1.80 to 2.17, keyword+lowercase 2.03 and 1.88,
standard+word_delimiter+lowercase 1.33 and 1.41. pinyin is not in the
benchmark and is not touched.
- check-build-hygiene.sh passes; clang-format 16 leaves the changed files
as they are; run-clang-tidy.sh reports nothing on the changed lines.
- Behavior changed:
- [x] No.
- [ ] Yes.
Every token and every exception is the same as before; only the call
an exception comes out of can be a later one (above).
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
