[ 
https://issues.apache.org/jira/browse/LUCENE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated LUCENE-6809:
-----------------------------------
    Labels: pull-request-available  (was: )

> DictionaryCompoundWordTokenFilter should respect minSubwordSize also for 
> fragments
> ----------------------------------------------------------------------------------
>
>                 Key: LUCENE-6809
>                 URL: https://issues.apache.org/jira/browse/LUCENE-6809
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: modules/analysis
>    Affects Versions: 5.3, 6.0
>            Reporter: Christian Winkler
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: LUCENE-6809.diff
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{DictionaryCompoundWordTokenFilter}} is very useful for building German 
> search indices. However this can also lead to ambiguities as words might be 
> extracted which have a completely different meaning. Most often this happens 
> if the remaining parts of the word are too small.
> Example: {{schwein}} (German for pig) contains {{wein}} (German for wine). 
> Even if {{minSubwordSize}} is set to {{4}}, {{wein}} gets extracted though 
> {{sch}} is shorter than 4 characters.
> We could solve this by requiring all word parts to be part of the dictionary 
> (at most 3), but this creates problems with compound words of more than three 
> nouns.
> Therefore we have built an alternate solution where {{minSubwordSize}} is 
> also applied to the rest of the fragments. We have tested this in several 
> (large) customer indices and it is working much better than before.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

Reply via email to