[
https://issues.apache.org/jira/browse/LUCENE-9929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17321313#comment-17321313
]
Jan Høydahl commented on LUCENE-9929:
-------------------------------------
We have a customer who reported a bug that the "oo" and "ao" foldings cause
issues.
A {{NorwegianNormalizationFilter}} naming would be misleading for such a new
filter, as these filters are not aiming to only do Norwegian folding (which
would simply be ae->æ, oe->ø, aa->å). The aim of {{Scandinavian*Filters}} is to
normalize all the Scandinavian special characters across the three languages,
i.e. æ==ä==ae, ø==ö==oe==oo, å==aa==ao. It actually does this pretty well, i.e.
by matching "Göteborg" even if you search on a Norwegian keyboard "gøteborg" or
if you only have a US keyboard and type "goeteborg", and most of this is safe
and unambigous. Except the slight variation in two-char folds, which we now
bring under the user's control.
We could of course consider a "foldings=<NO,SV,DK>" as an alternative to the
low-levelparameter. But I'm not sure if it makes sense to make one fixed choice
for Denmark and Sweden; besides, the "foldings" parameter also allows for
disabling all five two-char foldings if you are only interested in the æøåäö
foldings.
See PR for working code.
> Make ScandinavianNormalizationFilter configurable wrt foldings
> --------------------------------------------------------------
>
> Key: LUCENE-9929
> URL: https://issues.apache.org/jira/browse/LUCENE-9929
> Project: Lucene - Core
> Issue Type: Improvement
> Components: modules/analysis
> Reporter: Jan Høydahl
> Assignee: Jan Høydahl
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The ScandinavianNormalizationFilter applies foldings for aa, ao, ae, oe and
> oo. But all those five do not make sense for both Norwegian, Swedish and
> Danish. Implement an optional configuration option where users can select
> which of them to apply. I.e. for Norwegian, a user would then configure (in
> Solr):
> {code:java}
> <filter class="solr.ScandinavianNormalizationFilterFactory
> foldings="ae,oe,aa"/>
> {code}
> This would activate foldings for ae->æ, oe->ø, aa->å, but not oo->o and ao->a.
> The default will be to activate all five as before, so it will be backward
> compatible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
