Hi Naomi, I don't have a conclusive answer for you on this yet, but let me pick up on a few points.
First, the apostrophe is probably being handled through ignoring punctuation in the ICUCollationKeyFilterFactory. Alif isn't a diacritic but a letter, and its character properties would be handled as such, apparently also outside the scope of what the folding filter factory does unless it's tailored. >From the solrwiki, this looks like a helpful rule of thumb: "When To use a CharFilter vs a TokenFilter There are several pairs of CharFilters and TokenFilters that have related (ie: MappingCharFilter and ASCIIFoldingFilter) or nearly identical functionality (ie: PatternReplaceCharFilterFactory and PatternReplaceFilterFactory) and it may not always be obvious which is the best choice. The ultimate decision depends largely on what Tokenizer you are using, and whether you need to "out smart" it by preprocessing the stream of characters. For example, maybe you have a tokenizer such as StandardTokenizer and you are pretty happy with how it works overall, but you want to customize how some specific characters behave. In such a situation you could modify the rules and re-build your own tokenizer with javacc, but perhaps its easier to simply map some of the characters before tokenization with a CharFilter." Charles On Tue, May 15, 2012 at 2:47 PM, Naomi Dushay <ndus...@stanford.edu> wrote: > We are using the ICUFoldingFilterFactory with great success to fold > diacritics so searches with and without the diacritics get the same results. > > We recently discovered we have some Korean records that use an alif > diacritic instead of an apostrophe, and this diacritic is NOT getting > folded. Has anyone experienced this for alif or ayn characters? Do you > have a solution? > > > - Naomi > > -- > You received this message because you are subscribed to the Google Groups > "solrmarc-tech" group. > To post to this group, send email to solrmarc-t...@googlegroups.com. > To unsubscribe from this group, send email to > solrmarc-tech+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/solrmarc-tech?hl=en. > > -- *Charles L. Riley* *Catalog Librarian for Africana* *Sterling Memorial Library, Yale University* *<**zenodo...@gmail.com* <zenodo...@gmail.com>*>* *203-432-7566*
