Hi Erick,
Thanks for the reply.
Here is what the situation is:
Relevant portion of Solr Schema:
<field name="Content" type="text_general" indexed="false" stored="true"
required="true"/>
<field name="ContentSearch" type="text_general" indexed="true"
stored="false" multiValued="true"/>
<field name="ContentSearchStemming" type="text_stem" indexed="true"
stored="false" multiValued="true"/>
<copyField source="Content" dest="ContentSearch"/>
<copyField source="Content" dest="ContentSearchStemming"/>
<fieldType name="text_general" class="solr.TextField"
positionIncrementGap="100"> <analyzer type="index"> <tokenizer
class="solr.StandardTokenizerFactory"/> <filter
class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt"
enablePositionIncrements="true" /> <filter
class="solr.LowerCaseFilterFactory"/> </analyzer> <analyzer
type="query"> <tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" /> <filter
class="solr.LowerCaseFilterFactory"/> </analyzer> </fieldType>
<fieldType name="text_stem" class="solr.TextField" > <analyzer>
<tokenizer class="solr.WhitespaceTokenizerFactory"/> <filter
class="solr.SnowballPorterFilterFactory"/> </analyzer>
</fieldType>
When I am indexing a document, the content gets stored as is in the Content
field and gets copied over to ContentSearch and ContentSearchStemming for text
based search and stemming search respectively. So, the ContentSearchStemming
field does store the
stem/reduced form of the terms. I have checked this with the Luke as well as
the Admin Schema Browser --> Term Info. In the Admin
Analysis screen, I have tested and found that if I index the text "burning", it
gets reduced to and stored as "burn". So far so good.
Now in the UI,
lets say the user puts in the term "burn" and checks the stemming option. The
expectation is that since the user has specified stemming, the results should
be returned for the term "burn" as well as for all terms which has their stem
as "burn" i.e. burning, burned, burns, etc.
lets say the user puts in the term "burning" and checks the stemming option.
The expectation is that since the user has specified stemming, the results
should be returned for the term "burning" as well as for all terms which has
their stem as "burn" i.e. burn, burned, burns, etc.
The query that gets submitted to Solr: q=ContentSearchStemming:burning
>From Debug Info:
<str name="rawquerystring">ContentSearchStemming:burning</str>
<str name="querystring">ContentSearchStemming:burning</str>
<str name="parsedquery">ContentSearchStemming:burn</str>
<str name="parsedquery_toString">ContentSearchStemming:burn</str>
So, when the results are returned, I am only getting the hits highlighted with
the term "burn", though the same document contains terms like burning and
burns.
I thought that the stemming should work like this:
The stemming filter in the queryanalyzer chain would reduce the input word to
its stem. burning --> burn
The query component should scan through the terms and match those terms for
which it finds a match between the stem of the term with the stem of the input
term. burns --> burn (matches) burning --> burn
The first point is happening. But looks like it is executing the search for an
exact text based match with the stem "burn". Hence, burns or burned are not
getting returned.
Hope I was able to make myself clear.
---- On Fri, 28 Jun 2013 05:59:37 -0700 Erick Erickson [via Lucene]
<ml-node+s472066n4073901...@n3.nabble.com> wrote ----
First, this is for the Java version, I hope it extends to C#.
But in your configuration, when you're indexing the stemmer
should be storing the reduced form in the index. Then, when
searching, the search should be against the reduced term.
To check this, try
1> Using the Admin/Analysis page to see what gets stored
in your index and what your query is transformed to to
insure that you're getting what you expect.
If you want to get in deeper to the details, try
1> use, say, the TermsComponent or Admin/Schema Browser
or Luke to look in your index and see what's actually
there.
2> us &debug=query or Admin/Analysis to see what the query
actually looks like.
Both your use-cases should work fine just with reduction
_unless_ the particular word you look for doesn't happen to
trip the stemmer. By that I mean that since it's algorithmically
based, there may be some edge cases that seem like they
should be reduced that aren't. I don't know whether "fisherman"
would reduce to "fish" for instance.
So are you seeing things that really don't work as expected or
are you just working from the docs? Because I really don't
see why you wouldn't get what you want given your description.
Best
Erick
On Fri, Jun 28, 2013 at 2:33 AM, snkar <[hidden email]> wrote:
> We have a search system based on Solr using the Solrnet library in C#
which
> supports some advanced search features like Fuzzy, Synonym and Stemming.
> While all of these work, *the expectation from the Stemming Search seems
to
> be a combination of Stemming by reduction as well as stemming by expansion
> to cover grammatical variations on a word*. A use case will make it more
> clear:
>
> - a search for fish would also find fishing
> - a search for applied would also find applying, applies, and apply
>
> We had implemented Stemming using a CopyField with
> SnowballPorterFilterFactory. *As a result, when /searching for burning the
> results are returning for burning and burn/ but when /searching for burn
> the
> results are not returning for burning or burnt or burns/*
>
> Since all stemmers supported Lucene/Solr all use stemming by reduction, we
> are not sure on how to go about this. As per the Solr Wiki:
>
> > A related technology to stemming is lemmatization, which allows for
> > "stemming" by expansion, taking a root word and 'expanding' it to all
of
> > its various forms. Lemmatization can be used either at insertion time
or
> > at query time. Lucene/Solr does not have built-in support for
> > lemmatization but it can be simulated by using your own dictionaries
and
> > the SynonymFilterFactory
>
> We are not sure of exactly how to go about this in Solr. Any ideas.
>
> We were also thinking in terms of using some C# based stemmer/lemmatizer
> library to get the root of the word and using some public database like
> WordNet to extract the different grammatical variations of the stem and
> then
> send across all these terms for querying in Solr. We have not yet done a
> lot
> of research to figure out a stable C# stemmer/lemmatizer and a WordNet C#
> API, but seems like this will get too convoluted and it should have a way
> to
> be executed from within Solr.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Stemming-query-in-Solr-tp4073862.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
If you reply to this email, your message will be added to the discussion
below:
http://lucene.472066.n3.nabble.com/Stemming-query-in-Solr-tp4073862p4073901.html
To unsubscribe from Stemming query in Solr, click here.
NAML
--
View this message in context:
http://lucene.472066.n3.nabble.com/Stemming-query-in-Solr-tp4073862p4074283.html
Sent from the Solr - User mailing list archive at Nabble.com.
