Robert Muir created LUCENE-10363:
------------------------------------
Summary: JapaneseCompletionFilter messes up offsets
Key: LUCENE-10363
URL: https://issues.apache.org/jira/browse/LUCENE-10363
Project: Lucene - Core
Issue Type: Bug
Reporter: Robert Muir
It is a tokenfilter, tries to change offsets, so of course TestRandomChains
finds bugs in it:
{noformat}
NOTE: reproduce with: gradlew test --tests
TestRandomChains.testRandomChainsWithLargeStrings -Dtests.seed=E233A5FAC016E02
-Dtests.nightly=true -Dtests.slow=true -Dtests.locale=en-TV
-Dtests.timezone=Asia/Saigon -Dtests.asserts=true -Dtests.file.encoding=UTF-8
{noformat}
{noformat}
org.apache.lucene.analysis.tests.TestRandomChains > test suite's output saved
to
/home/rmuir/workspace/lucene/lucene/analysis/integration.tests/build/test-results/test_54/outputs/OUTPUT-org.apache.lucene.analysis.tests.TestRandomChains.txt,
copied below:
2> stage 0: lk<[1-3] +1> p<[6-7] +1> ngtoixtmldzsjz<[10-24] +1> uoq<[25-28]
+1> HANGUL<[28-28] +1> o<[29-30] +1> HANGUL<[31-31] +1> VulliPHsZzn<[32-43] +1>
2> stage 1: lk<[1-3] +1> 850000<[1-3] +0> p<[6-7] +1> 700000<[6-7] +0>
ngtoixtmldzsjz<[10-24] +1> 653543<[10-24] +0> uoq<[25-28] +1> 050000<[25-28]
+0> HANGUL<[28-28] +1> 565800<[28-28] +0> o<[29-30] +1> 000000<[29-30] +0>
HANGUL<[31-31] +1> 565800<[31-31] +0> VulliPHsZzn<[32-43] +1> 787460<[32-43] +0>
2> stage 2: ngtoixtmldzsjz 653543<[10-24] +0> 653543<[10-24] +1> 653543
uoq<[10-28] +0> uoq<[25-28] +1> uoq 050000<[25-28] +0> 050000<[25-28] +1>
050000 HANGUL<[25-28] +0> HANGUL<[28-28] +1> HANGUL 565800<[28-28] +0>
565800<[28-28] +1> 565800 o<[28-30] +0> o<[29-30] +1> o 000000<[29-30] +0>
000000<[29-30] +1> 000000 HANGUL<[29-31] +0> HANGUL<[31-31] +1> HANGUL
565800<[31-31] +0> 565800<[31-31] +1> 565800 VulliPHsZzn<[31-43] +0>
VulliPHsZzn<[32-43] +1>
2> last stage: ngtoixtmldzsjz<[10-24] +1> ngtoixtmldzsjz 653543<[10-24] +0>
653543<[10-24] +1> 653543 uoq<[10-28] +0> uoq<[25-28] +1> uoq 050000<[25-28]
+1> 050000<[25-28] +1> 050000 HANGUL<[25-28] +1> HANGUL<[28-28] +1> HANGUL
565800<[28-28] +0> 565800<[28-28] +1> 565800 o<[28-30] +0> o<[29-30] +1> o
000000<[29-30] +0> 000000<[29-30] +1> 000000 HANGUL<[29-31] +0> HANGUL<[31-31]
+1> HANGUL 565800<[31-31] +1> 565800<[31-31] +1> 565800 VulliPHsZzn<[31-43] +0>
2> TEST FAIL: useCharFilter=true text='[lk[-.p|) ngtoixtmldzsjz uoqao
aVulliPHsZzn wxsk'
2> Exception from random analyzer:
2> charfilters=
2> org.apache.lucene.analysis.pattern.PatternReplaceCharFilter(a, <HANGUL>,
java.io.StringReader@5b3b54eb)
2> tokenizer=
2>
org.apache.lucene.analysis.classic.ClassicTokenizer(org.apache.lucene.util.AttributeFactory$1@e29311e9)
2> filters=
2>
org.apache.lucene.analysis.phonetic.DaitchMokotoffSoundexFilter(ValidatingTokenFilter@32a6de77
term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,
true)
2>
org.apache.lucene.analysis.shingle.ShingleFilter(ValidatingTokenFilter@3d044414
term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,
q)
2>
Conditional:org.apache.lucene.analysis.ja.JapaneseCompletionFilter(OneTimeWrapper@435207ec
term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,reading=null,reading
(en)=null,pronunciation=null,pronunciation (en)=null, INDEX)
> java.lang.IllegalStateException: last stage: inconsistent endOffset at
pos=19: 31 vs 43; token=565800 VulliPHsZzn
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]