Yonik, thank you for your reply. When I don't use PreserveOriginal = 1 for
WordDelimiterFilterFactory, the query "ain't" is parsed as "ain t" and no match
is found in this case too. If I remove ' from the query, then I can get
results. I used the analysis tool and see the term ain't is processed as "ain
t", and get matches when the title includes "ain't". But I got no result when
using ain't query with dismax.
The debug output looks like:
(NON-MATCH) Failure to meet condition(s) of required/prohibited clause(s)
+(long_description:"ain t"^2.0 | name:"ain t"^3.0 | search_keywords:"ain
t")~0.1 (long_description:save^2.0 | name:save^3.0 |
search_keywords:saved)~0.1) ()
Below is my configuration for text field type.
<fieldType name="text" class="solr.TextField" positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="false"/>
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt" enablePositionIncrements="true" />
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0"
splitOnCaseChange="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPorterFilterFactory"
protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<!--<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
ignoreCase="true" expand="true"/>-->
<filter class="solr.StopFilterFactory" ignoreCase="true"
words="stopwords.txt"/>
<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0"
splitOnCaseChange="0"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.EnglishPorterFilterFactory"
protected="protwords.txt"/>
<filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
</analyzer>
</fieldType>
I get results back when I tried to use solr.LowerCaseTokenizerFactory instead
of solr.WhitespaceTokenizerFactory. However, the concern here is this might
reduce the quality of relevant search. Does anyone have a better idea on what
to try next? Thanks!
Wen
-----Original Message-----
From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley
Sent: Thursday, March 11, 2010 10:51 AM
To: solr-user@lucene.apache.org
Subject: Re: dismax and WordDelimiterFilterFactory with PreserveOriginal = 1
On Thu, Mar 11, 2010 at 1:07 PM, Ya-Wen Hsu <y...@eline.com> wrote:
> Hi all,
>
> I'm facing the same issue as previous post here:
> http://www.mail-archive.com/solr-user@lucene.apache.org/msg19511.html. Since
> no one answers this post, I thought I'll ask again. In my case, I use below
> setting for index
> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
> generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0"
> splitOnCaseChange="0" preserveOriginal="1"/>
> and
> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
> generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0"
> splitOnCaseChange="0" preserveOriginal="1"/> for query.
>
> When I use query with word "ain't", no result is returned. When I turned on
> the logging, I found the word is interpreted as "(ain't ain) t".
The problem is preserving the original in the query analyzer - try
removing that. And if you aren't doing prefix or wildcard queries,
preserveOriginal doesn't buy you anything but wasted index space.
It's the same issue of why you can't generate and catenate at the same
time with the query parser.
-Yonik
http://www.lucidimagination.com
