Hi..
Could you tell me if changing default similarity to custom implementation
will require me to rebuild the index? Or will it be used only query time?
thanks,
Sandeep
On 31 Jan 2013 13:55, "Felipe Lahti" <fla...@thoughtworks.com> wrote:
> So, it depends of your business requirement, right? If a document has
> matches in more searchable fields, at least for me, this document is more
> important than other document that has less matches.
>
> Example:
> Put this in your schema:
> <similarity class="com.your.namespace.NoIDFSimilarity" />
>
> And create a class in your classpath of your Solr:
>
> package com.your.namespace;
>
> import org.apache.lucene.search.similarities.DefaultSimilarity;
>
> public class NoIDFSimilarity extends DefaultSimilarity {
>
> @Override
>
> public float idf(long docFreq, long numDocs) {
>
> return 1;
>
> }
>
> }
>
>
> It will "neutralize" the idf (which is the rarity of term).
>
>
>
>
>
>
> On Thu, Jan 31, 2013 at 5:31 AM, Sandeep Mestry <sanmes...@gmail.com>
> wrote:
>
> > Thanks Felipe..
> > Can you point me an example please?
> >
> > Also forgive me but if a document has matches in more searchable fields
> > then should it not rank higher?
> >
> > Thanks,
> > Sandeep
> > On 30 Jan 2013 19:30, "Felipe Lahti" <fla...@thoughtworks.com> wrote:
> >
> > > If you compare the first and last document scores you will see that the
> > > last one matches more fields than first one. So, you maybe thinking
> why?
> > > The first doc only matches "contributions" field and the last matches a
> > > bunch of fields so if you want to have behave more like (<str
> > > name="qf">series_title^500 title^100 description^15 contribution</str>)
> > you
> > > have to override the method of DefaultSimilarity.
> > >
> > >
> > > On Wed, Jan 30, 2013 at 4:12 PM, Sandeep Mestry <sanmes...@gmail.com>
> > > wrote:
> > >
> > > > I have pasted it below and it is slightly variant from the dismax
> > > > configuration I have mentioned above as I was playing with all sorts
> of
> > > > boost values, however it looks more lie below:
> > > >
> > > > <str name="c208c2ca-4270-27b8-e040-a8c00409063a">
> > > > 2675.7844 = (MATCH) sum of: 2675.7844 = (MATCH) max plus 0.01 times
> > > others
> > > > of: 2675.7844 = (MATCH) weight(contributions:news in 63298)
> > > > [DefaultSimilarity], result of: 2675.7844 = score(doc=63298,freq=1.0
> =
> > > > termFreq=1.0 ), product of: 0.004495774 = queryWeight, product of:
> > > > 14.530705 = idf(docFreq=14, maxDocs=11282414) 3.093982E-4 = queryNorm
> > > > 595177.7 = fieldWeight in 63298, product of: 1.0 = tf(freq=1.0), with
> > > freq
> > > > of: 1.0 = termFreq=1.0 14.530705 = idf(docFreq=14, maxDocs=11282414)
> > > > 40960.0 = fieldNorm(doc=63298)
> > > > </str>
> > > > <str name="c208c2a9-66bc-27b8-e040-a8c00409063a">
> > > > 2317.297 = (MATCH) sum of: 2317.297 = (MATCH) max plus 0.01 times
> > others
> > > > of: 2317.297 = (MATCH) weight(contributions:news in 9826415)
> > > > [DefaultSimilarity], result of: 2317.297 =
> score(doc=9826415,freq=3.0 =
> > > > termFreq=3.0 ), product of: 0.004495774 = queryWeight, product of:
> > > > 14.530705 = idf(docFreq=14, maxDocs=11282414) 3.093982E-4 = queryNorm
> > > > 515439.0 = fieldWeight in 9826415, product of: 1.7320508 =
> > tf(freq=3.0),
> > > > with freq of: 3.0 = termFreq=3.0 14.530705 = idf(docFreq=14,
> > > > maxDocs=11282414) 20480.0 = fieldNorm(doc=9826415)
> > > > </str>
> > > > <str name="c208c2aa-1806-27b8-e040-a8c00409063a">
> > > > 2140.6274 = (MATCH) sum of: 2140.6274 = (MATCH) max plus 0.01 times
> > > others
> > > > of: 2140.6274 = (MATCH) weight(contributions:news in 9882325)
> > > > [DefaultSimilarity], result of: 2140.6274 =
> score(doc=9882325,freq=1.0
> > =
> > > > termFreq=1.0 ), product of: 0.004495774 = queryWeight, product of:
> > > > 14.530705 = idf(docFreq=14, maxDocs=11282414) 3.093982E-4 = queryNorm
> > > > 476142.16 = fieldWeight in 9882325, product of: 1.0 = tf(freq=1.0),
> > with
> > > > freq of: 1.0 = termFreq=1.0 14.530705 = idf(docFreq=14,
> > maxDocs=11282414)
> > > > 32768.0 = fieldNorm(doc=9882325)
> > > > </str>
> > > > <str name="c208c2b0-5165-27b8-e040-a8c00409063a">
> > > > 1605.4707 = (MATCH) sum of: 1605.4707 = (MATCH) max plus 0.01 times
> > > others
> > > > of: 1605.4707 = (MATCH) weight(contributions:news in 220007)
> > > > [DefaultSimilarity], result of: 1605.4707 =
> score(doc=220007,freq=1.0 =
> > > > termFreq=1.0 ), product of: 0.004495774 = queryWeight, product of:
> > > > 14.530705 = idf(docFreq=14, maxDocs=11282414) 3.093982E-4 = queryNorm
> > > > 357106.62 = fieldWeight in 220007, product of: 1.0 = tf(freq=1.0),
> with
> > > > freq of: 1.0 = termFreq=1.0 14.530705 = idf(docFreq=14,
> > maxDocs=11282414)
> > > > 24576.0 = fieldNorm(doc=220007)
> > > > </str>
> > > > <str name="c208c2cc-d01b-27b8-e040-a8c00409063a">
> > > > 1605.4707 = (MATCH) sum of: 1605.4707 = (MATCH) max plus 0.01 times
> > > others
> > > > of: 1605.4707 = (MATCH) weight(contributions:news in 241151)
> > > > [DefaultSimilarity], result of: 1605.4707 =
> score(doc=241151,freq=1.0 =
> > > > termFreq=1.0 ), product of: 0.004495774 = queryWeight, product of:
> > > > 14.530705 = idf(docFreq=14, maxDocs=11282414) 3.093982E-4 = queryNorm
> > > > 357106.62 = fieldWeight in 241151, product of: 1.0 = tf(freq=1.0),
> with
> > > > freq of: 1.0 = termFreq=1.0 14.530705 = idf(docFreq=14,
> > maxDocs=11282414)
> > > > 24576.0 = fieldNorm(doc=241151)
> > > > </str>
> > > > </lst>
> > > > <str name="otherQuery">id:c208c2b4-1b3e-27b8-e040-a8c00409063a</str>
> > > > <lst name="explainOther">
> > > > <str name="*c208c2b4-1b3e-27b8-e040-a8c00409063a*"> <!-- this should
> > rank
> > > > higher -->
> > > > 6.5742764 = (MATCH) sum of: 6.5742764 = (MATCH) max plus 0.01 times
> > > others
> > > > of: 3.304414 = (MATCH) weight(description:news^25.0 in 967895)
> > > > [DefaultSimilarity], result of: 3.304414 = score(doc=967895,freq=1.0
> =
> > > > termFreq=1.0 ), product of: 0.042727955 = queryWeight, product of:
> > 25.0 =
> > > > boost 5.5240083 = idf(docFreq=122362, maxDocs=11282414) 3.093982E-4 =
> > > > queryNorm 77.33611 = fieldWeight in 967895, product of: 1.0 =
> > > tf(freq=1.0),
> > > > with freq of: 1.0 = termFreq=1.0 5.5240083 = idf(docFreq=122362,
> > > > maxDocs=11282414) 14.0 = fieldNorm(doc=967895) 5.913381 = (MATCH)
> > > > weight(pg_series_title:news^50.0 in 967895) [DefaultSimilarity],
> result
> > > of:
> > > > 5.913381 = score(doc=967895,freq=1.0 = termFreq=1.0 ), product of:
> > > > 0.080834694 = queryWeight, product of: 50.0 = boost 5.2252855 =
> > > > idf(docFreq=164961, maxDocs=11282414) 3.093982E-4 = queryNorm 73.154
> =
> > > > fieldWeight in 967895, product of: 1.0 = tf(freq=1.0), with freq of:
> > 1.0
> > > =
> > > > termFreq=1.0 5.2252855 = idf(docFreq=164961, maxDocs=11282414) 14.0 =
> > > > fieldNorm(doc=967895) 0.18680073 = (MATCH)
> > weight(p_programme_title:news
> > > in
> > > > 967895) [DefaultSimilarity], result of: 0.18680073 =
> > > > score(doc=967895,freq=1.0 = termFreq=1.0 ), product of: 0.002031815 =
> > > > queryWeight, product of: 6.5669904 = idf(docFreq=43120,
> > maxDocs=11282414)
> > > > 3.093982E-4 = queryNorm 91.93787 = fieldWeight in 967895, product of:
> > > 1.0 =
> > > > tf(freq=1.0), with freq of: 1.0 = termFreq=1.0 6.5669904 =
> > > > idf(docFreq=43120, maxDocs=11282414) 14.0 = fieldNorm(doc=967895)
> > > 6.464123
> > > > = (MATCH) weight(pg_series_title_ci:news^500.0 in 967895)
> > > > [DefaultSimilarity], result of: 6.464123 = score(doc=967895,freq=1.0
> =
> > > > termFreq=1.0 ), product of: 0.99999696 = queryWeight, product of:
> > 500.0 =
> > > > boost 6.4641423 = idf(docFreq=47791, maxDocs=11282414) 3.093982E-4 =
> > > > queryNorm 6.4641423 = fieldWeight in 967895, product of: 1.0 =
> > > > tf(freq=1.0), with freq of: 1.0 = termFreq=1.0 6.4641423 =
> > > > idf(docFreq=47791, maxDocs=11282414) 1.0 = fieldNorm(doc=967895)
> > > 1.6107484
> > > > = (MATCH) weight(title_ci:news^100.0 in 967895) [DefaultSimilarity],
> > > result
> > > > of: 1.6107484 = score(doc=967895,freq=1.0 = termFreq=1.0 ), product
> of:
> > > > 0.22324038 = queryWeight, product of: 100.0 = boost 7.2153096 =
> > > > idf(docFreq=22548, maxDocs=11282414) 3.093982E-4 = queryNorm
> 7.2153096
> > =
> > > > fieldWeight in 967895, product of: 1.0 = tf(freq=1.0), with freq of:
> > 1.0
> > > =
> > > > termFreq=1.0 7.2153096 = idf(docFreq=22548, maxDocs=11282414) 1.0 =
> > > > fieldNorm(doc=967895)
> > > > </str>
> > > >
> > > >
> > > > On 30 January 2013 17:55, Felipe Lahti <fla...@thoughtworks.com>
> > wrote:
> > > >
> > > > > Let me see if I understood your problem:
> > > > >
> > > > > By your first e-mail I think you are worried about the returned
> order
> > > of
> > > > > documents from Solr. Is that correct? If yes, as I said before it's
> > not
> > > > > only the boosting that influence the order of returned documents.
> > > There's
> > > > > term frequency, IDF(inverse document frequency)... If I understood
> > > > > correctly by your first e-mail, you are interested in get rid of
> IDF.
> > > So
> > > > > for that, you can create a NoIDFSimilarity class to override the
> > > default
> > > > > similarity.
> > > > >
> > > > > Can you paste here the score calculation for one document?
> > > > >
> > > > >
> > > > > On Wed, Jan 30, 2013 at 2:06 PM, Sandeep Mestry <
> sanmes...@gmail.com
> > > > >wrote:
> > > > >
> > > > >> (Sorry for in complete reply in my previous mail, didn't know
> Ctrl F
> > > > sends
> > > > >> an email in Gmail.. ;-))
> > > > >>
> > > > >> Thanks Felipe, yes I have seen that and my requirement falls for
> > > > >>
> > > > >> How can I make exact-case matches score higher
> > > > >>
> > > > >> Example: a query of "Penguin" should score documents containing
> > > > "Penguin"
> > > > >> higher than docs containing "penguin".
> > > > >>
> > > > >> The general strategy is to index the content twice, using
> different
> > > > fields
> > > > >> with different fieldTypes (and different analyzers associated with
> > > those
> > > > >> fieldTypes). One analyzer will contain a lowercase filter for
> > > > >> case-insensitive matches, and one will preserve case for
> exact-case
> > > > >> matches.
> > > > >>
> > > > >> Use copyField <http://wiki.apache.org/solr/SchemaXml#copyField>
> > > > commands
> > > > >> in
> > > > >>
> > > > >> the schema to index a single input field multiple times.
> > > > >>
> > > > >> Once the content is indexed into multiple fields that are analyzed
> > > > >> differently, query across both
> > > > >> fields<
> http://wiki.apache.org/solr/SolrRelevancyFAQ#multiFieldQuery
> > >
> > > > >>
> > > > >> .
> > > > >>
> > > > >> I have added a case insensitive field too to match the exact
> matches
> > > > >> higher, however the result is not even considering the matches in
> > > field
> > > > -
> > > > >> forget the exact matching part.
> > > > >>
> > > > >> And I have tried the debugQuery option as mentioned in my previous
> > > mail,
> > > > >> and I have also posted the parsed queries. From the debug query, I
> > see
> > > > >> that
> > > > >> field boosted with lesser factor (contribution) is still resulting
> > > > higher
> > > > >> than the one with higher boost factor (series_title).
> > > > >>
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Sandeep
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On 30 January 2013 16:02, Sandeep Mestry <sanmes...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > Thanks Felipe, yes I have seen that and my requirement somewhere
> > > falls
> > > > >> for
> > > > >> >
> > > > >> >
> > > > >> > On 30 January 2013 15:53, Felipe Lahti <fla...@thoughtworks.com
> >
> > > > wrote:
> > > > >> >
> > > > >> >> Hi Sandeep,
> > > > >> >>
> > > > >> >> Quick answer is that not only the boost that you define in your
> > > > >> >> requestHandler is taken to calculate the score of each
> document.
> > > > There
> > > > >> are
> > > > >> >> others factors that contribute to score calculation. You can
> > take a
> > > > >> look
> > > > >> >> here about http://wiki.apache.org/solr/SolrRelevancyFAQ. Also,
> > you
> > > > can
> > > > >> >> see
> > > > >> >> using debugQuery=true the score calculation for each document
> > > > returned.
> > > > >> >>
> > > > >> >> Let me know you need something else.
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> On Wed, Jan 30, 2013 at 1:13 PM, Sandeep Mestry <
> > > sanmes...@gmail.com
> > > > >
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > Hi All,
> > > > >> >> >
> > > > >> >> > I'm facing an issue in relevancy calculation by dismax query
> > > > parser.
> > > > >> >> > The boost factor applied does not work as expected in certain
> > > cases
> > > > >> when
> > > > >> >> > the keyword is generic and by generic I mean, if the keyword
> is
> > > > >> >> appearing
> > > > >> >> > many times in the document as well as in the index.
> > > > >> >> >
> > > > >> >> > I have parser configuration as below:
> > > > >> >> >
> > > > >> >> > <requestHandler name="querydismax"
> class="solr.SearchHandler" >
> > > > >> >> > <lst name="defaults">
> > > > >> >> > <str name="defType">edismax</str>
> > > > >> >> > <str name="echoParams">explicit</str>
> > > > >> >> > <float name="tie">0.01</float>
> > > > >> >> > <str name="qf">series_title^500 title^100
> > > > description^15
> > > > >> >> > contribution</str>
> > > > >> >> > <str name="pf">series_title^200</str>
> > > > >> >> > <int name="ps">0</int>
> > > > >> >> > <str name="q.alt">*:*</str>
> > > > >> >> > </lst>
> > > > >> >> > </requestHandler>
> > > > >> >> >
> > > > >> >> > As you can see above, I'd expect the documents containing the
> > > > matches
> > > > >> >> for
> > > > >> >> > series title should rank higher than the ones in
> contribution.
> > > > >> >> >
> > > > >> >> > This works well, if I type in a query like 'wonderworld'
> which
> > > is a
> > > > >> less
> > > > >> >> > occurring term and the series titles rank higher. But, if I
> > type
> > > > in a
> > > > >> >> > keyword like 'news' which is the most common term in the
> > index, I
> > > > get
> > > > >> >> hits
> > > > >> >> > in contributions even though I have lots of documents having
> > word
> > > > >> news
> > > > >> >> in
> > > > >> >> > series title.
> > > > >> >> >
> > > > >> >> > The field definition is as below:
> > > > >> >> >
> > > > >> >> > <field name="series_title" type="text_wc" indexed="true"
> > > > >> stored="true"
> > > > >> >> > multiValued="false" />
> > > > >> >> > <field name="title" type="text_wc" indexed="true"
> stored="true"
> > > > >> >> > multiValued="false" />
> > > > >> >> > <field name="description" type="text_wc" indexed="true"
> > > > stored="true"
> > > > >> >> > multiValued="false" />
> > > > >> >> > <field name="contribution" type="text" indexed="true"
> > > stored="true"
> > > > >> >> > multiValued="true" />
> > > > >> >> >
> > > > >> >> > <fieldType name="text" class="solr.TextField"
> > > > >> positionIncrementGap="100"
> > > > >> >> > compressThreshold="10">
> > > > >> >> > <analyzer type="index">
> > > > >> >> > <tokenizer
> > > > class="solr.WhitespaceTokenizerFactory"/>
> > > > >> >> > <filter
> class="solr.WordDelimiterFilterFactory"
> > > > >> >> > generateWordParts="1" generateNumberParts="1"
> catenateWords="1"
> > > > >> >> > catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/>
> > > > >> >> > <filter class="solr.LowerCaseFilterFactory"/>
> > > > >> >> > </analyzer>
> > > > >> >> > <analyzer type="query">
> > > > >> >> > <tokenizer
> > > > class="solr.WhitespaceTokenizerFactory"/>
> > > > >> >> > <filter
> class="solr.WordDelimiterFilterFactory"
> > > > >> >> > generateWordParts="1" generateNumberParts="1"
> catenateWords="0"
> > > > >> >> > catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/>
> > > > >> >> > <filter class="solr.LowerCaseFilterFactory"/>
> > > > >> >> > </analyzer>
> > > > >> >> > </fieldType>
> > > > >> >> >
> > > > >> >> > <fieldType name="text_wc" class="solr.TextField"
> > > > >> >> positionIncrementGap="100"
> > > > >> >> > >
> > > > >> >> > <analyzer type="index">
> > > > >> >> > <tokenizer
> > > > class="solr.WhitespaceTokenizerFactory"/>
> > > > >> >> > <filter
> class="solr.WordDelimiterFilterFactory"
> > > > >> >> > stemEnglishPossessive="0" generateWordParts="1"
> > > > >> generateNumberParts="1"
> > > > >> >> > catenateWords="1" catenateNumbers="1" catenateAll="1"
> > > > >> >> splitOnCaseChange="1"
> > > > >> >> > splitOnNumerics="0" preserveOriginal="1" />
> > > > >> >> > <filter class="solr.LowerCaseFilterFactory"/>
> > > > >> >> > </analyzer>
> > > > >> >> > <analyzer type="query">
> > > > >> >> > <tokenizer
> > > > class="solr.WhitespaceTokenizerFactory"/>
> > > > >> >> > <filter
> class="solr.WordDelimiterFilterFactory"
> > > > >> >> > stemEnglishPossessive="0" generateWordParts="1"
> > > > >> generateNumberParts="1"
> > > > >> >> > catenateWords="1" catenateNumbers="1" catenateAll="1"
> > > > >> >> splitOnCaseChange="1"
> > > > >> >> > splitOnNumerics="0" preserveOriginal="1" />
> > > > >> >> > <filter class="solr.LowerCaseFilterFactory"/>
> > > > >> >> > </analyzer>
> > > > >> >> > </fieldType>
> > > > >> >> >
> > > > >> >> > I have tried debugging and when I use query term news, I see
> > that
> > > > >> >> matches
> > > > >> >> > for contributions are ranked higher than series title. The
> > parsed
> > > > >> >> queries
> > > > >> >> > look like below:
> > > > >> >> > (Note that I have edited the query as in reality I have lot
> of
> > > > fields
> > > > >> >> that
> > > > >> >> > are searchable and I have only mentioned the fields
> containing
> > > text
> > > > >> >> data -
> > > > >> >> > rest all contain uuids)
> > > > >> >> >
> > > > >> >> > <str name="parsedquery">
> > > > >> >> > (+DisjunctionMaxQuery((description:news^15.0 |
> > title:news^100.0 |
> > > > >> >> > contributions:news | series_title:news^500.0)~0.01) () () ()
> ()
> > > ()
> > > > ()
> > > > >> >> () ()
> > > > >> >> > () () () () () () () () () () () () () () () () () () ()
> > > > ())/no_coord
> > > > >> >> > </str>
> > > > >> >> > <str name="parsedquery_toString">
> > > > >> >> > +(description:news^15 | title:news^100.0 |
> contributions:news |
> > > > >> >> > series_title:news^500.0)~0.01 () () () () () () () () () ()
> ()
> > ()
> > > > ()
> > > > >> ()
> > > > >> >> ()
> > > > >> >> > () () () () () () () () () () () () ()
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > Could you guide me in right direction please?
> > > > >> >> >
> > > > >> >> > Many Thanks,
> > > > >> >> > Sandeep
> > > > >> >> >
> > > > >> >>
> > > > >> >>
> > > > >> >>
> > > > >> >> --
> > > > >> >> Felipe Lahti
> > > > >> >> Consultant Developer - ThoughtWorks Porto Alegre
> > > > >> >>
> > > > >> >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Felipe Lahti
> > > > > Consultant Developer - ThoughtWorks Porto Alegre
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Felipe Lahti
> > > Consultant Developer - ThoughtWorks Porto Alegre
> > >
> >
>
>
>
> --
> Felipe Lahti
> Consultant Developer - ThoughtWorks Porto Alegre
>
