The problem with french is slightly different but leads to somewhat the
same problems: there are no ascii folding configured currently.
This leads to the same intitle problems where it can't find words with
diacritics when diacritics are omitted in the query.
For french putting ascii folding before the stemmer is certainly a bad
idea and we should imo do ascii folding after the stemmer (possibly
using preserve original).
Le 10/08/2016 à 16:52, Trey Jones a écrit :
I'm less sure that re-ordering would do the right thing for French.
Presumably the French stemmer knows about accented characters and uses
them. We should test and make sure. Maybe we need custom folding only
for "unusual" accents in any given language (they are all unusual for
English).
We can test French the same way we tested English, though, and be sure.
—Trey
Trey Jones
Software Engineer, Discovery
Wikimedia Foundation
On Wed, Aug 10, 2016 at 10:48 AM, David Causse <[email protected]
<mailto:[email protected]>> wrote:
Thanks Trey!
this will certainly greatly improve the intitle keyword as it uses
the field with stems for filtering and hopefully will find pages
that were ignored because of this filter ordering (e.g.
intitle:louys can't find User:Louÿs currently).
I think I'll do the same for French which suffers from the same
problem. IMO we should continue to work on this for other
languages while we try to switch from asciifolding (latin letters
only) to icu folding.
We may require some guidance on some languages where diacritics
removal can be counter productive and maybe blacklist some letters
(e.g. for finnish: is it appropriate to fold Ä or Ö for example?)
Note on accent folding: cirrus tries to always prefer exact
matches. Searching for élément should always prefer élément over
element. Users that prefer exact matches can always force cirrus
to discard stems by wrapping the word in double quotes, e.g.
"élément".
Le 10/08/2016 à 16:07, Trey Jones a écrit :
David and I had a discussion about moving ascii-folding to come
before stemming on English Wikipedia. It seemed like a good idea,
but we decided we should run some tests before implementing it,
just to be sure.
Turns out it is a good idea!
Much more detail:
https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/Re-Ordering_Stemming_and_Ascii-Folding_on_English_Wikipedia
<https://www.mediawiki.org/wiki/User:TJones_%28WMF%29/Notes/Re-Ordering_Stemming_and_Ascii-Folding_on_English_Wikipedia>
We won't deploy it until we deploy BM25 later in the year, since
it requires a full re-index of English Wikipedia, as does BM25.
That's something we should only do once.
—Trey
Trey Jones
Software Engineer, Discovery
Wikimedia Foundation
_______________________________________________
discovery mailing list
[email protected] <mailto:[email protected]>
https://lists.wikimedia.org/mailman/listinfo/discovery
<https://lists.wikimedia.org/mailman/listinfo/discovery>
_______________________________________________ discovery mailing
list [email protected]
<mailto:[email protected]>
https://lists.wikimedia.org/mailman/listinfo/discovery
<https://lists.wikimedia.org/mailman/listinfo/discovery>
_______________________________________________
discovery mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/discovery
_______________________________________________
discovery mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/discovery
