Thanks Shawn. So to recap: - Every "match" must be found after entire chain, not in the middle of the chain. - Suggested: <index> and <query> chain should be the same.
In my situation, if I make both of them the same, the result may be misleading because it will also match other records that have the same partial string. But your suggestion is wonderful. Thank you very much. -----Original Message----- From: Shawn Heisey [mailto:s...@elyograg.org] Sent: 2011年11月23日 12:04 下午 To: solr-user@lucene.apache.org Subject: Re: If search matches index in the middle of filter chain, will result return? On 11/22/2011 7:54 PM, Ellery Leung wrote: > I am searching for an email called: off...@officeofficeoffice.com. If I > search any text under 20 characters, result will be returned. But when I > search the whole string: off...@officeofficeoffice.com, no result return. > > As you all see in the schema in "index" part, when I search the whole > string, it will match the index chain before NGramFilterFactory. But after > NGram, no result found. > > Here are my questions: > - Is this behavior normal? I'm pretty sure that your query must match after the entire analyzer chain is done. I would expect that behavior to be normal. > - In order to get "off...@officeofficeoffice.com", does it mean > that I have to make the maxGramSize larger (like 70)? If you were to increase the maxGramSize to 70, you would get a match in this case, but your index might get a lot larger, depending on what's in your source data. That's probably not the right approach, though. In general, you want to have your index and query analyzer chains exactly the same. There are some exceptions, but I don't think the NGram filter is one of them. The synonym filter and WordDelimiterFilter are examples where it is expected that your index and query analyzer chains will be different. Add the NGram and CommonGram filters to the query chain, and everything should start working. If you were to go with a single analyzer for both like the following, I think it would start working. You wouldn't even need to reindex, since you wouldn't be changing the index analyzer. <fieldType name="substring_search" class="solr.TextField" positionIncrementGap="100"> <analyzer> <charFilter class="solr.MappingCharFilterFactory" mapping="../../filters/filter-mappings.txt"/> <charFilter class="solr.HTMLStripCharFilterFactory" /> <tokenizer class="solr.KeywordTokenizerFactory"/> <filter class="solr.ASCIIFoldingFilterFactory"/> <filter class="solr.TrimFilterFactory" /> <filter class="solr.LowerCaseFilterFactory" /> <filter class="solr.CommonGramsFilterFactory" words="../../filters/stopwords.txt" ignoreCase="true"/> <filter class="solr.NGramFilterFactory" minGramSize="1" maxGramSize="20"/> <filter class="solr.RemoveDuplicatesTokenFilterFactory" /> </analyzer> </fieldType> Regarding your NGram filter, I would actually increase the minGramSize to at least 2 and decrease the maxGramSize to something like 10 or 15, then reindex. An additional note: CommonGrams may not be all that useful unless you are indexing large numbers of huge documents, like entire books. This particular fieldType is not suitable for full text anyway, since it uses KeywordTokenizer. Consider removing CommonGrams from this fieldType and reindexing. Unless you are dealing with large amounts of text, consider removing it from the entire schema. If you do remove it, it's usually not a good idea to replace it with a StopFilter. The index size reduction found in stopword removal is not usually worth the potential loss of recall. Be prepared to test all reasonable analyzer combinations, rather than taking my word for it. After reading the Hathi Trust blog, I tried CommonGrams on my own index. It actually made things slower, not faster. My typical document is only a few thousand bytes of metadata. The Hathi Trust is indexing millions of full-length books. Thanks, Shawn
