If you compare the first and last document scores you will see that the last one matches more fields than first one. So, you maybe thinking why? The first doc only matches "contributions" field and the last matches a bunch of fields so if you want to have behave more like (<str name="qf">series_title^500 title^100 description^15 contribution</str>) you have to override the method of DefaultSimilarity.
On Wed, Jan 30, 2013 at 4:12 PM, Sandeep Mestry <sanmes...@gmail.com> wrote: > I have pasted it below and it is slightly variant from the dismax > configuration I have mentioned above as I was playing with all sorts of > boost values, however it looks more lie below: > > <str name="c208c2ca-4270-27b8-e040-a8c00409063a"> > 2675.7844 = (MATCH) sum of: 2675.7844 = (MATCH) max plus 0.01 times others > of: 2675.7844 = (MATCH) weight(contributions:news in 63298) > [DefaultSimilarity], result of: 2675.7844 = score(doc=63298,freq=1.0 = > termFreq=1.0 ), product of: 0.004495774 = queryWeight, product of: > 14.530705 = idf(docFreq=14, maxDocs=11282414) 3.093982E-4 = queryNorm > 595177.7 = fieldWeight in 63298, product of: 1.0 = tf(freq=1.0), with freq > of: 1.0 = termFreq=1.0 14.530705 = idf(docFreq=14, maxDocs=11282414) > 40960.0 = fieldNorm(doc=63298) > </str> > <str name="c208c2a9-66bc-27b8-e040-a8c00409063a"> > 2317.297 = (MATCH) sum of: 2317.297 = (MATCH) max plus 0.01 times others > of: 2317.297 = (MATCH) weight(contributions:news in 9826415) > [DefaultSimilarity], result of: 2317.297 = score(doc=9826415,freq=3.0 = > termFreq=3.0 ), product of: 0.004495774 = queryWeight, product of: > 14.530705 = idf(docFreq=14, maxDocs=11282414) 3.093982E-4 = queryNorm > 515439.0 = fieldWeight in 9826415, product of: 1.7320508 = tf(freq=3.0), > with freq of: 3.0 = termFreq=3.0 14.530705 = idf(docFreq=14, > maxDocs=11282414) 20480.0 = fieldNorm(doc=9826415) > </str> > <str name="c208c2aa-1806-27b8-e040-a8c00409063a"> > 2140.6274 = (MATCH) sum of: 2140.6274 = (MATCH) max plus 0.01 times others > of: 2140.6274 = (MATCH) weight(contributions:news in 9882325) > [DefaultSimilarity], result of: 2140.6274 = score(doc=9882325,freq=1.0 = > termFreq=1.0 ), product of: 0.004495774 = queryWeight, product of: > 14.530705 = idf(docFreq=14, maxDocs=11282414) 3.093982E-4 = queryNorm > 476142.16 = fieldWeight in 9882325, product of: 1.0 = tf(freq=1.0), with > freq of: 1.0 = termFreq=1.0 14.530705 = idf(docFreq=14, maxDocs=11282414) > 32768.0 = fieldNorm(doc=9882325) > </str> > <str name="c208c2b0-5165-27b8-e040-a8c00409063a"> > 1605.4707 = (MATCH) sum of: 1605.4707 = (MATCH) max plus 0.01 times others > of: 1605.4707 = (MATCH) weight(contributions:news in 220007) > [DefaultSimilarity], result of: 1605.4707 = score(doc=220007,freq=1.0 = > termFreq=1.0 ), product of: 0.004495774 = queryWeight, product of: > 14.530705 = idf(docFreq=14, maxDocs=11282414) 3.093982E-4 = queryNorm > 357106.62 = fieldWeight in 220007, product of: 1.0 = tf(freq=1.0), with > freq of: 1.0 = termFreq=1.0 14.530705 = idf(docFreq=14, maxDocs=11282414) > 24576.0 = fieldNorm(doc=220007) > </str> > <str name="c208c2cc-d01b-27b8-e040-a8c00409063a"> > 1605.4707 = (MATCH) sum of: 1605.4707 = (MATCH) max plus 0.01 times others > of: 1605.4707 = (MATCH) weight(contributions:news in 241151) > [DefaultSimilarity], result of: 1605.4707 = score(doc=241151,freq=1.0 = > termFreq=1.0 ), product of: 0.004495774 = queryWeight, product of: > 14.530705 = idf(docFreq=14, maxDocs=11282414) 3.093982E-4 = queryNorm > 357106.62 = fieldWeight in 241151, product of: 1.0 = tf(freq=1.0), with > freq of: 1.0 = termFreq=1.0 14.530705 = idf(docFreq=14, maxDocs=11282414) > 24576.0 = fieldNorm(doc=241151) > </str> > </lst> > <str name="otherQuery">id:c208c2b4-1b3e-27b8-e040-a8c00409063a</str> > <lst name="explainOther"> > <str name="*c208c2b4-1b3e-27b8-e040-a8c00409063a*"> <!-- this should rank > higher --> > 6.5742764 = (MATCH) sum of: 6.5742764 = (MATCH) max plus 0.01 times others > of: 3.304414 = (MATCH) weight(description:news^25.0 in 967895) > [DefaultSimilarity], result of: 3.304414 = score(doc=967895,freq=1.0 = > termFreq=1.0 ), product of: 0.042727955 = queryWeight, product of: 25.0 = > boost 5.5240083 = idf(docFreq=122362, maxDocs=11282414) 3.093982E-4 = > queryNorm 77.33611 = fieldWeight in 967895, product of: 1.0 = tf(freq=1.0), > with freq of: 1.0 = termFreq=1.0 5.5240083 = idf(docFreq=122362, > maxDocs=11282414) 14.0 = fieldNorm(doc=967895) 5.913381 = (MATCH) > weight(pg_series_title:news^50.0 in 967895) [DefaultSimilarity], result of: > 5.913381 = score(doc=967895,freq=1.0 = termFreq=1.0 ), product of: > 0.080834694 = queryWeight, product of: 50.0 = boost 5.2252855 = > idf(docFreq=164961, maxDocs=11282414) 3.093982E-4 = queryNorm 73.154 = > fieldWeight in 967895, product of: 1.0 = tf(freq=1.0), with freq of: 1.0 = > termFreq=1.0 5.2252855 = idf(docFreq=164961, maxDocs=11282414) 14.0 = > fieldNorm(doc=967895) 0.18680073 = (MATCH) weight(p_programme_title:news in > 967895) [DefaultSimilarity], result of: 0.18680073 = > score(doc=967895,freq=1.0 = termFreq=1.0 ), product of: 0.002031815 = > queryWeight, product of: 6.5669904 = idf(docFreq=43120, maxDocs=11282414) > 3.093982E-4 = queryNorm 91.93787 = fieldWeight in 967895, product of: 1.0 = > tf(freq=1.0), with freq of: 1.0 = termFreq=1.0 6.5669904 = > idf(docFreq=43120, maxDocs=11282414) 14.0 = fieldNorm(doc=967895) 6.464123 > = (MATCH) weight(pg_series_title_ci:news^500.0 in 967895) > [DefaultSimilarity], result of: 6.464123 = score(doc=967895,freq=1.0 = > termFreq=1.0 ), product of: 0.99999696 = queryWeight, product of: 500.0 = > boost 6.4641423 = idf(docFreq=47791, maxDocs=11282414) 3.093982E-4 = > queryNorm 6.4641423 = fieldWeight in 967895, product of: 1.0 = > tf(freq=1.0), with freq of: 1.0 = termFreq=1.0 6.4641423 = > idf(docFreq=47791, maxDocs=11282414) 1.0 = fieldNorm(doc=967895) 1.6107484 > = (MATCH) weight(title_ci:news^100.0 in 967895) [DefaultSimilarity], result > of: 1.6107484 = score(doc=967895,freq=1.0 = termFreq=1.0 ), product of: > 0.22324038 = queryWeight, product of: 100.0 = boost 7.2153096 = > idf(docFreq=22548, maxDocs=11282414) 3.093982E-4 = queryNorm 7.2153096 = > fieldWeight in 967895, product of: 1.0 = tf(freq=1.0), with freq of: 1.0 = > termFreq=1.0 7.2153096 = idf(docFreq=22548, maxDocs=11282414) 1.0 = > fieldNorm(doc=967895) > </str> > > > On 30 January 2013 17:55, Felipe Lahti <fla...@thoughtworks.com> wrote: > > > Let me see if I understood your problem: > > > > By your first e-mail I think you are worried about the returned order of > > documents from Solr. Is that correct? If yes, as I said before it's not > > only the boosting that influence the order of returned documents. There's > > term frequency, IDF(inverse document frequency)... If I understood > > correctly by your first e-mail, you are interested in get rid of IDF. So > > for that, you can create a NoIDFSimilarity class to override the default > > similarity. > > > > Can you paste here the score calculation for one document? > > > > > > On Wed, Jan 30, 2013 at 2:06 PM, Sandeep Mestry <sanmes...@gmail.com > >wrote: > > > >> (Sorry for in complete reply in my previous mail, didn't know Ctrl F > sends > >> an email in Gmail.. ;-)) > >> > >> Thanks Felipe, yes I have seen that and my requirement falls for > >> > >> How can I make exact-case matches score higher > >> > >> Example: a query of "Penguin" should score documents containing > "Penguin" > >> higher than docs containing "penguin". > >> > >> The general strategy is to index the content twice, using different > fields > >> with different fieldTypes (and different analyzers associated with those > >> fieldTypes). One analyzer will contain a lowercase filter for > >> case-insensitive matches, and one will preserve case for exact-case > >> matches. > >> > >> Use copyField <http://wiki.apache.org/solr/SchemaXml#copyField> > commands > >> in > >> > >> the schema to index a single input field multiple times. > >> > >> Once the content is indexed into multiple fields that are analyzed > >> differently, query across both > >> fields<http://wiki.apache.org/solr/SolrRelevancyFAQ#multiFieldQuery> > >> > >> . > >> > >> I have added a case insensitive field too to match the exact matches > >> higher, however the result is not even considering the matches in field > - > >> forget the exact matching part. > >> > >> And I have tried the debugQuery option as mentioned in my previous mail, > >> and I have also posted the parsed queries. From the debug query, I see > >> that > >> field boosted with lesser factor (contribution) is still resulting > higher > >> than the one with higher boost factor (series_title). > >> > >> > >> Thanks, > >> > >> Sandeep > >> > >> > >> > >> > >> On 30 January 2013 16:02, Sandeep Mestry <sanmes...@gmail.com> wrote: > >> > >> > Thanks Felipe, yes I have seen that and my requirement somewhere falls > >> for > >> > > >> > > >> > On 30 January 2013 15:53, Felipe Lahti <fla...@thoughtworks.com> > wrote: > >> > > >> >> Hi Sandeep, > >> >> > >> >> Quick answer is that not only the boost that you define in your > >> >> requestHandler is taken to calculate the score of each document. > There > >> are > >> >> others factors that contribute to score calculation. You can take a > >> look > >> >> here about http://wiki.apache.org/solr/SolrRelevancyFAQ. Also, you > can > >> >> see > >> >> using debugQuery=true the score calculation for each document > returned. > >> >> > >> >> Let me know you need something else. > >> >> > >> >> > >> >> > >> >> On Wed, Jan 30, 2013 at 1:13 PM, Sandeep Mestry <sanmes...@gmail.com > > > >> >> wrote: > >> >> > >> >> > Hi All, > >> >> > > >> >> > I'm facing an issue in relevancy calculation by dismax query > parser. > >> >> > The boost factor applied does not work as expected in certain cases > >> when > >> >> > the keyword is generic and by generic I mean, if the keyword is > >> >> appearing > >> >> > many times in the document as well as in the index. > >> >> > > >> >> > I have parser configuration as below: > >> >> > > >> >> > <requestHandler name="querydismax" class="solr.SearchHandler" > > >> >> > <lst name="defaults"> > >> >> > <str name="defType">edismax</str> > >> >> > <str name="echoParams">explicit</str> > >> >> > <float name="tie">0.01</float> > >> >> > <str name="qf">series_title^500 title^100 > description^15 > >> >> > contribution</str> > >> >> > <str name="pf">series_title^200</str> > >> >> > <int name="ps">0</int> > >> >> > <str name="q.alt">*:*</str> > >> >> > </lst> > >> >> > </requestHandler> > >> >> > > >> >> > As you can see above, I'd expect the documents containing the > matches > >> >> for > >> >> > series title should rank higher than the ones in contribution. > >> >> > > >> >> > This works well, if I type in a query like 'wonderworld' which is a > >> less > >> >> > occurring term and the series titles rank higher. But, if I type > in a > >> >> > keyword like 'news' which is the most common term in the index, I > get > >> >> hits > >> >> > in contributions even though I have lots of documents having word > >> news > >> >> in > >> >> > series title. > >> >> > > >> >> > The field definition is as below: > >> >> > > >> >> > <field name="series_title" type="text_wc" indexed="true" > >> stored="true" > >> >> > multiValued="false" /> > >> >> > <field name="title" type="text_wc" indexed="true" stored="true" > >> >> > multiValued="false" /> > >> >> > <field name="description" type="text_wc" indexed="true" > stored="true" > >> >> > multiValued="false" /> > >> >> > <field name="contribution" type="text" indexed="true" stored="true" > >> >> > multiValued="true" /> > >> >> > > >> >> > <fieldType name="text" class="solr.TextField" > >> positionIncrementGap="100" > >> >> > compressThreshold="10"> > >> >> > <analyzer type="index"> > >> >> > <tokenizer > class="solr.WhitespaceTokenizerFactory"/> > >> >> > <filter class="solr.WordDelimiterFilterFactory" > >> >> > generateWordParts="1" generateNumberParts="1" catenateWords="1" > >> >> > catenateNumbers="1" catenateAll="0" splitOnCaseChange="1"/> > >> >> > <filter class="solr.LowerCaseFilterFactory"/> > >> >> > </analyzer> > >> >> > <analyzer type="query"> > >> >> > <tokenizer > class="solr.WhitespaceTokenizerFactory"/> > >> >> > <filter class="solr.WordDelimiterFilterFactory" > >> >> > generateWordParts="1" generateNumberParts="1" catenateWords="0" > >> >> > catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> > >> >> > <filter class="solr.LowerCaseFilterFactory"/> > >> >> > </analyzer> > >> >> > </fieldType> > >> >> > > >> >> > <fieldType name="text_wc" class="solr.TextField" > >> >> positionIncrementGap="100" > >> >> > > > >> >> > <analyzer type="index"> > >> >> > <tokenizer > class="solr.WhitespaceTokenizerFactory"/> > >> >> > <filter class="solr.WordDelimiterFilterFactory" > >> >> > stemEnglishPossessive="0" generateWordParts="1" > >> generateNumberParts="1" > >> >> > catenateWords="1" catenateNumbers="1" catenateAll="1" > >> >> splitOnCaseChange="1" > >> >> > splitOnNumerics="0" preserveOriginal="1" /> > >> >> > <filter class="solr.LowerCaseFilterFactory"/> > >> >> > </analyzer> > >> >> > <analyzer type="query"> > >> >> > <tokenizer > class="solr.WhitespaceTokenizerFactory"/> > >> >> > <filter class="solr.WordDelimiterFilterFactory" > >> >> > stemEnglishPossessive="0" generateWordParts="1" > >> generateNumberParts="1" > >> >> > catenateWords="1" catenateNumbers="1" catenateAll="1" > >> >> splitOnCaseChange="1" > >> >> > splitOnNumerics="0" preserveOriginal="1" /> > >> >> > <filter class="solr.LowerCaseFilterFactory"/> > >> >> > </analyzer> > >> >> > </fieldType> > >> >> > > >> >> > I have tried debugging and when I use query term news, I see that > >> >> matches > >> >> > for contributions are ranked higher than series title. The parsed > >> >> queries > >> >> > look like below: > >> >> > (Note that I have edited the query as in reality I have lot of > fields > >> >> that > >> >> > are searchable and I have only mentioned the fields containing text > >> >> data - > >> >> > rest all contain uuids) > >> >> > > >> >> > <str name="parsedquery"> > >> >> > (+DisjunctionMaxQuery((description:news^15.0 | title:news^100.0 | > >> >> > contributions:news | series_title:news^500.0)~0.01) () () () () () > () > >> >> () () > >> >> > () () () () () () () () () () () () () () () () () () () > ())/no_coord > >> >> > </str> > >> >> > <str name="parsedquery_toString"> > >> >> > +(description:news^15 | title:news^100.0 | contributions:news | > >> >> > series_title:news^500.0)~0.01 () () () () () () () () () () () () > () > >> () > >> >> () > >> >> > () () () () () () () () () () () () () > >> >> > > >> >> > > >> >> > Could you guide me in right direction please? > >> >> > > >> >> > Many Thanks, > >> >> > Sandeep > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Felipe Lahti > >> >> Consultant Developer - ThoughtWorks Porto Alegre > >> >> > >> > > >> > > >> > > > > > > > > -- > > Felipe Lahti > > Consultant Developer - ThoughtWorks Porto Alegre > > > -- Felipe Lahti Consultant Developer - ThoughtWorks Porto Alegre
