Nice! Thanks! -----Original Message----- From: Em [mailto:mailformailingli...@yahoo.de] Sent: Thursday, April 21, 2011 9:23 AM To: solr-user@lucene.apache.org Subject: RE: stemming filter analyzers, any favorites?
As far as I know Lucene does not store an inverted index per field, so no, it would not double the size of the index. However, it could influence the score a little bit. For example: If both stemmers reduce "schools" to "school" and you are searching for "all schools in america" the term "school" has more weight to the resulting score, since it definitly occurs in two fields which consist of nearly the same value. To reduce this effect you could write your own queryParser which creates a disjunctionMaxQuery consisting of two boolean queries and a tie-break of 0 - so only the better scoring stemmed-field contributes to the total score of your document. Regards, Em Robert Petersen-3 wrote: > > Adding another field with another stemmer and searching both??? Wow never > thought of doing that. I guess that doesn't really double the size of > your index tho because all the terms are almost the same right? Let me > look into that. I'll raise the other issue in a separate thread and > thanks. > > -----Original Message----- > From: Em [mailto:mailformailingli...@yahoo.de] > Sent: Thursday, April 21, 2011 1:55 AM > To: solr-user@lucene.apache.org > Subject: RE: stemming filter analyzers, any favorites? > > Hi Robert, > > we often ran into the same issue with stemmers. This is why we created > more > than one field, each field with different stemmers. It adds some overhead > but worked quite well. > > Regarding your off-topic-question: > Look at the debugging-output of your searches. Sometimes you configured > your > tools, especially the WDF, wrong and the queryParser creates an unexpected > result which leads to unmatched but still relevant documents. > > Please, show us your debugging-output and the field-definition so that we > can provide you some help! > > Regards, > Em > > > Robert Petersen-3 wrote: >> >> I have been doing that, and for Bags example the trailing 's' is not >> being >> removed by the Kstemmer so if indexing the word bags and searching on bag >> you get no matches. Why wouldn't the trailing 's' get stemmed off? >> Kstemmer is dictionary based so bags isn't in the dictionary? That >> trailing 's' should always be dropped no? That seems like it would be >> better, we don't want to make synonyms for basic use cases like this. I >> fear I will have to return to the Porter stemmer. Are there other better >> ones is my main question. >> >> Off topic secondary question: sometimes I am puzzled by the output of the >> analysis page. It seems like there should be a match, but I don't get >> the >> results during a search that I'd expect... >> >> Like in the case if the WordDelimiterFilterFactory splits up a term into >> a >> bunch of terms before the K-stemmer is applied, sometimes if the matching >> term is in position two of the final analysis but the searcher had the >> partial term just alone and so thereby in position 1 in the analysis >> stack >> then when searching there wasn't a match. Am I reading this correctly? >> Is that right or should that match and I am misreading my analysis >> output? >> >> Thanks! >> >> Robi >> >> PS I have a category named Bags and am catching flack for it not coming >> up in a search for bag. hah >> PPS the term is not in protwords.txt >> >> >> com.lucidimagination.solrworks.analysis.LucidKStemFilterFactory >> {protected=protwords.txt} >> term position 1 >> term text bags >> term type word >> source start,end 0,4 >> payload >> >> >> -----Original Message----- >> From: Erick Erickson [mailto:erickerick...@gmail.com] >> Sent: Wednesday, April 20, 2011 10:55 AM >> To: solr-user@lucene.apache.org >> Subject: Re: stemming filter analyzers, any favorites? >> >> You can get a better sense of exactly what tranformations occur when >> if you look at the analysis page (be sure to check the "verbose" >> checkbox). >> >> I'm surprised that "bags" doesn't match "bag", what does the analysis >> page say? >> >> Best >> Erick >> >> On Wed, Apr 20, 2011 at 1:44 PM, Robert Petersen <rober...@buy.com> >> wrote: >>> Stemming filter analyzers... anyone have any favorites for particular >>> search domains? Just wondering what people are using. I'm using Lucid >>> K Stemmer and having issues. Seems like it misses a lot of common >>> stems. We went to that because of excessively loose matches on the >>> solr.PorterStemFilterFactory >>> >>> >>> I understand K Stemmer is a dictionary based stemmer. Seems to me like >>> it is missing a lot of common stem reductions. Ie Bags does not match >>> Bag in our searches. >>> >>> Here is my analyzer stack: >>> >>> <fieldType name="text" class="solr.TextField" >>> positionIncrementGap="100"> >>> <analyzer type="index"> >>> <tokenizer >>> class="solr.WhitespaceTokenizerFactory"/> >>> <filter >>> class="solr.SynonymFilterFactory" synonyms="index_synonyms.txt" >>> ignoreCase="true" expand="true"/> >>> <filter class="solr.StopFilterFactory" >>> ignoreCase="true" words="stopwords.txt"/> >>> <filter class="solr.WordDelimiterFilterFactory" >>> generateWordParts="1" >>> generateNumberParts="1" >>> catenateWords="1" >>> catenateNumbers="1" >>> catenateAll="1" >>> preserveOriginal="1" >>> /> <filter >>> class="solr.LowerCaseFilterFactory"/> >>> >>> <filter >>> class="com.lucidimagination.solrworks.analysis.LucidKStemFilterFactory" >>> protected="protwords.txt"/> >>> <filter >>> class="solr.RemoveDuplicatesTokenFilterFactory"/> >>> </analyzer> >>> <analyzer type="query"> >>> <tokenizer >>> class="solr.WhitespaceTokenizerFactory"/> >>> <filter >>> class="solr.SynonymFilterFactory" synonyms="query_synonyms.txt" >>> ignoreCase="true" expand="true"/> >>> <filter class="solr.StopFilterFactory" >>> ignoreCase="true" words="stopwords.txt"/> >>> <filter class="solr.WordDelimiterFilterFactory" >>> generateWordParts="1" >>> generateNumberParts="1" >>> catenateWords="1" >>> catenateNumbers="1" >>> catenateAll="1" >>> preserveOriginal="1" >>> /> <filter >>> class="solr.LowerCaseFilterFactory"/> >>> >>> <filter >>> class="com.lucidimagination.solrworks.analysis.LucidKStemFilterFactory" >>> protected="protwords.txt"/> >>> <filter >>> class="solr.RemoveDuplicatesTokenFilterFactory"/> >>> </analyzer> >>> </fieldType> >>> >> > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/stemming-filter-analyzers-any-favorites-tp2844517p2846336.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- View this message in context: http://lucene.472066.n3.nabble.com/stemming-filter-analyzers-any-favorites-tp2844517p2847585.html Sent from the Solr - User mailing list archive at Nabble.com.
