I'm less sure that re-ordering would do the right thing for French. Presumably the French stemmer knows about accented characters and uses them. We should test and make sure. Maybe we need custom folding only for "unusual" accents in any given language (they are all unusual for English).
We can test French the same way we tested English, though, and be sure. —Trey Trey Jones Software Engineer, Discovery Wikimedia Foundation On Wed, Aug 10, 2016 at 10:48 AM, David Causse <[email protected]> wrote: > Thanks Trey! > > this will certainly greatly improve the intitle keyword as it uses the > field with stems for filtering and hopefully will find pages that were > ignored because of this filter ordering (e.g. intitle:louys can't find > User:Louÿs currently). > > I think I'll do the same for French which suffers from the same problem. > IMO we should continue to work on this for other languages while we try to > switch from asciifolding (latin letters only) to icu folding. > > We may require some guidance on some languages where diacritics removal > can be counter productive and maybe blacklist some letters (e.g. for > finnish: is it appropriate to fold Ä or Ö for example?) > > Note on accent folding: cirrus tries to always prefer exact matches. > Searching for élément should always prefer élément over element. Users that > prefer exact matches can always force cirrus to discard stems by wrapping > the word in double quotes, e.g. "élément". > > Le 10/08/2016 à 16:07, Trey Jones a écrit : > > David and I had a discussion about moving ascii-folding to come before > stemming on English Wikipedia. It seemed like a good idea, but we decided > we should run some tests before implementing it, just to be sure. > > Turns out it is a good idea! > > Much more detail: > https://www.mediawiki.org/wiki/User:TJones_(WMF)/Notes/ > Re-Ordering_Stemming_and_Ascii-Folding_on_English_Wikipedia > > We won't deploy it until we deploy BM25 later in the year, since it > requires a full re-index of English Wikipedia, as does BM25. That's > something we should only do once. > > —Trey > > Trey Jones > Software Engineer, Discovery > Wikimedia Foundation > > > _______________________________________________ > discovery mailing > [email protected]https://lists.wikimedia.org/mailman/listinfo/discovery > > > > _______________________________________________ > discovery mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/discovery > >
_______________________________________________ discovery mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/discovery
