Maybe rerankqparserplugin?
On Aug 12, 2016 11:54, "John Bickerstaff" <j...@johnbickerstaff.com> wrote:
> @Hossman -- thanks again.
>
> I've made the following change and so far things look good. I couldn't see
> debug or find results for what I put in for $func, so I just removed it,
> but making modifications as you suggested appears to be working.
>
> Including the actual line from my endpoint XML in case this thread helps
> someone else...
>
> <str name="q">{!boost defType=synonym_edismax qf='title' synonyms='true'
> synonyms.originalBoost='2.5' synonyms.synonymBoost='1.1' bf='' bq=''
> v=$q}</str>
>
> On Fri, Aug 12, 2016 at 12:09 PM, John Bickerstaff <
> j...@johnbickerstaff.com
> > wrote:
>
> > Thanks! I'll check it out.
> >
> > On Fri, Aug 12, 2016 at 12:05 PM, Susheel Kumar <susheel2...@gmail.com>
> > wrote:
> >
> >> Not exactly sure what you are looking from chaining the results but
> >> similar
> >> functionality is available in Streaming expressions where result of
> inner
> >> expressions are passed to outer expressions and so on
> >> https://cwiki.apache.org/confluence/display/solr/Streaming+Expressions
> >>
> >> HTH
> >> Susheel
> >>
> >> On Fri, Aug 12, 2016 at 1:08 PM, John Bickerstaff <
> >> j...@johnbickerstaff.com>
> >> wrote:
> >>
> >> > Hossman - many thanks again for your comprehensive and very helpful
> >> answer!
> >> >
> >> > All,
> >> >
> >> > I am (possibly mis-remembering) reading something about being able to
> >> pass
> >> > the results of one query to another query... Essentially "chaining"
> >> result
> >> > sets.
> >> >
> >> > I have looked in docs and can't find anything on a quick search -- I
> may
> >> > have been reading about the Re-Ranking feature, which doesn't help me
> (I
> >> > know because I just tried and it seems to return all results anyway,
> >> just
> >> > re-ranking the number specified in the reRankDocs flag...)
> >> >
> >> > Is there a way to (cleanly) send the results of one query to another
> >> query
> >> > for further processing? Essentially, pass ONLY the results (including
> >> an
> >> > empty set of results) to another query for processing?
> >> >
> >> > thanks...
> >> >
> >> > On Thu, Aug 11, 2016 at 6:19 PM, John Bickerstaff <
> >> > j...@johnbickerstaff.com>
> >> > wrote:
> >> >
> >> > > Thanks!
> >> > >
> >> > > To answer your questions, while I digest the rest of that
> >> information...
> >> > >
> >> > > I'm using the hon-lucene-synonyms.5.0.4.jar from here:
> >> > > https://github.com/healthonnet/hon-lucene-synonyms
> >> > >
> >> > > The config looks like this - and IIRC, is simply a copy from the
> >> > > recommended cofig on the site mentioned above.
> >> > >
> >> > > <queryParser name="synonym_edismax" class="com.github.healthonnet.
> >> > search.
> >> > > SynonymExpandingExtendedDismaxQParserPlugin">
> >> > > <!-- You can define more than one synonym analyzer in the
> >> following
> >> > > list.
> >> > > For example, you might have one set of synonyms for
> English,
> >> one
> >> > > for French,
> >> > > one for Spanish, etc.
> >> > > -->
> >> > > <lst name="synonymAnalyzers">
> >> > > <!-- Name your analyzer something useful, e.g. "analyzer_en",
> >> > > "analyzer_fr", "analyzer_es", etc.
> >> > > If you only have one, the name doesn't matter (hence
> >> > > "myCoolAnalyzer").
> >> > > -->
> >> > > <lst name="myCoolAnalyzer">
> >> > > <!-- We recommend a PatternTokenizerFactory that tokenizes
> >> based
> >> > > on whitespace and quotes.
> >> > > This seems to work best with most people's synonym
> files.
> >> > > For details, read the discussion here:
> >> > > http://github.com/healthonnet/hon-lucene-synonyms/issues/26
> >> > > -->
> >> > > <lst name="tokenizer">
> >> > > <str name="class">solr.PatternTokenizerFactory</str>
> >> > > <str name="pattern"><![CDATA[(?:\s|\")+]]></str>
> >> > > </lst>
> >> > > <!-- The ShingleFilterFactory outputs synonyms of multiple
> >> token
> >> > > lengths (e.g. unigrams, bigrams, trigrams, etc.).
> >> > > The default here is to assume you don't have any
> synonyms
> >> > > longer than 4 tokens.
> >> > > You can tweak this depending on what your synonyms look
> >> > like.
> >> > > E.g. if you only have unigrams, you can remove
> >> > > it entirely, and if your synonyms are up to 7 tokens in
> >> > > length, you should set the maxShingleSize to 7.
> >> > > -->
> >> > > <lst name="filter">
> >> > > <str name="class">solr.ShingleFilterFactory</str>
> >> > > <str name="outputUnigramsIfNoShingles">true</str>
> >> > > <str name="outputUnigrams">true</str>
> >> > > <str name="minShingleSize">2</str>
> >> > > <str name="maxShingleSize">4</str>
> >> > > </lst>
> >> > > <!-- This is where you set your synonym file. For the unit
> >> tests
> >> > > and "Getting Started" examples, we use example_synonym_file.txt.
> >> > > This plugin will work best if you keep expand set to
> true
> >> > and
> >> > > have all your synonyms comma-separated (rather than =>-separated).
> >> > > -->
> >> > > <lst name="filter">
> >> > > <str name="class">solr.SynonymFilterFactory</str>
> >> > > <str name="tokenizerFactory">solr.
> >> > KeywordTokenizerFactory</str>
> >> > > <str name="synonyms">example_synonym_file.txt</str>
> >> > > <str name="expand">true</str>
> >> > > <str name="ignoreCase">true</str>
> >> > > </lst>
> >> > > </lst>
> >> > > </lst>
> >> > > </queryParser>
> >> > >
> >> > >
> >> > >
> >> > > On Thu, Aug 11, 2016 at 6:01 PM, Chris Hostetter <
> >> > hossman_luc...@fucit.org
> >> > > > wrote:
> >> > >
> >> > >>
> >> > >> : First let me say that this is very possibly the "x - y problem"
> so
> >> let
> >> > >> me
> >> > >> : state up front what my ultimate need is -- then I'll ask about
> the
> >> > >> thing I
> >> > >> : imagine might help... which, of course, is heavily biased in the
> >> > >> direction
> >> > >> : of my experience coding Java and writing SQL...
> >> > >>
> >> > >> Thank you so much for asking your question this way!
> >> > >>
> >> > >> Right off the bat, the background you've provided seems
> supicious...
> >> > >>
> >> > >> : I have a piece of a query that calculates a score based on a
> >> > "weighting"
> >> > >> ...
> >> > >> : The specific line is this:
> >> > >> : <str name="bf">product(field(category_weight),20)</str>
> >> > >> :
> >> > >> : What I just realized is that when I query Solr for a string that
> >> has
> >> > NO
> >> > >> : matches in the entire corpus, I still get a slew of results
> because
> >> > >> EVERY
> >> > >> : doc has the weighting value in the category_weight field - and
> >> > therefore
> >> > >> : every doc gets some score.
> >> > >>
> >> > >> ...that is *NOT* how dismax and edisamx normally work.
> >> > >>
> >> > >> While both the "bf" abd "bq" params result in "additive" boosting,
> >> and
> >> > the
> >> > >> implementation of that "additive boost" comes from adding new
> >> optional
> >> > >> clauses to the top level BooleanQuery that is executed, that only
> >> > happens
> >> > >> after the "main" query (from your "q" param) is added to that top
> >> level
> >> > >> BooleanQuery as a "mandaory" clause.
> >> > >>
> >> > >> So, for example, "bf=true()" and "bq=*:*" should match & boost
> every
> >> > doc,
> >> > >> but with the techprducts configs/data these requests still don't
> >> match
> >> > >> anything...
> >> > >>
> >> > >> /select?defType=edismax&q=bogus&bf=true()&bq=*:*&debug=query
> >> > >> /select?defType=dismax&q=bogus&bf=true()&bq=*:*&debug=query
> >> > >>
> >> > >> ...and if you look at the debug output, the parsed queries shows
> that
> >> > the
> >> > >> "bogus" part of the query is mandatory...
> >> > >>
> >> > >> +DisjunctionMaxQuery((text:bogus)) MatchAllDocsQuery(*:*)
> >> > >> FunctionQuery(const(true))
> >> > >>
> >> > >> (i didn't use "pf" in that example, but the effect is the same, the
> >> "pf"
> >> > >> based clauses are optional, while the "qf" based clauses are
> >> mandatory)
> >> > >>
> >> > >> If you compare that example to your debug output, you'll notice a
> >> > >> difference in structure -- it's a bit hard to see in your example,
> >> but
> >> > if
> >> > >> you simplify your qf, pf, and q fields it should be more obvious,
> but
> >> > >> AFAICT the "main" parts of your query are getting wrapped in an
> extra
> >> > >> layer of parents (ie: an extra BooleanQuery) which is *not*
> >> mandatory in
> >> > >> the top level query ... i don't see *any* mandatory clauses in your
> >> top
> >> > >> level BooleanQuery, which is why any match on a bf or bq function
> is
> >> > >> enough to cause a document to match.
> >> > >>
> >> > >> I suspect the reason your parsed query structure is so diff has to
> do
> >> > with
> >> > >> this...
> >> > >>
> >> > >> : <str name="defType">synonym_edismax</str>>
> >> > >>
> >> > >>
> >> > >> 1) how exactly is "synonym_edismax" defined in your solrconfig.xml?
> >> > >> 2) what QParserPlugin are you using to implement that?
> >> > >>
> >> > >> I suspect whatever QParserPlugin you are using has a bug in it :)
> >> > >>
> >> > >>
> >> > >> If you can't fix the bug, one possibile workaround would be to
> >> abandon
> >> > bf
> >> > >> and bq params completely, and instead wrap the query it produces in
> >> in a
> >> > >> {!boost} parser with whatever function you want (using functions
> like
> >> > >> sum() or prod() to combine multiple functions, and query() to
> >> > incorporate
> >> > >> your current bq param). Doing this will require chanign how you
> >> specify
> >> > >> you input (example below) and it will result in *multiplicitive*
> >> boosts
> >> > --
> >> > >> so your scores will be much diff, and you will likely have to
> adjust
> >> > your
> >> > >> constants, but: 1) multiplicitive boosts are almost always what
> >> people
> >> > >> *really* want anyway; 2) it will ensure the boosts are only applied
> >> for
> >> > >> things matching your main query, no matter how that query parser
> >> works
> >> > or
> >> > >> what bugs it has.
> >> > >>
> >> > >> Example of using {!boost} to wrap an arbitrary other parser...
> >> > >>
> >> > >> instead of...
> >> > >> defType=foofoo
> >> > >> q=barbarbar
> >> > >>
> >> > >> use...
> >> > >> q={!boost b=$func defType=foofoo v=$qq}
> >> > >> qq=barbarbar
> >> > >> func=sum(something,somethingelse)
> >> > >>
> >> > >> https://cwiki.apache.org/confluence/display/solr/Other+Parsers
> >> > >> https://cwiki.apache.org/confluence/display/solr/Function+Queries
> >> > >>
> >> > >>
> >> > >>
> >> > >>
> >> > >> :
> >> > >> : What I would like is to return zero results if there is no match
> >> for
> >> > the
> >> > >> : querystring. My collection is small enough that I don't care if
> >> the
> >> > >> actual
> >> > >> : calculation runs on each doc (although that's wasteful) -- I just
> >> > don't
> >> > >> : want to see results come back for zero matches to the querystring
> >> > >> :
> >> > >> : (The /select endpoint does this of course, but my custom endpoint
> >> > >> includes
> >> > >> : this "weighting" piece and therefore returns every doc in the
> >> corpus
> >> > >> : because they all have the weighting.
> >> > >> :
> >> > >> : ====================
> >> > >> : Enter my imagined solution... The potential X-Y problem...
> >> > >> : ====================
> >> > >> :
> >> > >> : So - given that I come from a programming background, I
> immediately
> >> > >> start
> >> > >> : thinking of an if statement ...
> >> > >> :
> >> > >> : if(some_score_for_the_primary_search_string) {
> >> > >> : run_the_category_weight_calculation;
> >> > >> : } else {
> >> > >> : do_NOT_run_category_weight_calc;
> >> > >> : }
> >> > >> :
> >> > >> :
> >> > >> : Another way of thinking of it would be something like the "WHERE"
> >> > >> clause in
> >> > >> : SQL...
> >> > >> :
> >> > >> : run_category_weight_calculation WHERE "searchstring" is found
> in
> >> the
> >> > >> : document, not otherwise.
> >> > >> :
> >> > >> : I'm aware that things could be handled in the client-side of my
> web
> >> > app,
> >> > >> : but if possible, I'd like the interface to SOLR to be as clean as
> >> > >> possible,
> >> > >> : and massage incoming SOLR data as little as possible.
> >> > >> :
> >> > >> : In other words, do NOT return any docs if the querystring (and
> any
> >> > >> : synonyms) match zero docs.
> >> > >> :
> >> > >> : Here is the endpoint XML for the query. I've highlighted the
> >> specific
> >> > >> line
> >> > >> : that is causing the unintended results...
> >> > >> :
> >> > >> :
> >> > >> : <requestHandler name="/foo" class="solr.SearchHandler">
> >> > >> : <!-- default values for query parameters can be specified,
> >> these
> >> > >> : will be overridden by parameters in the request
> >> > >> : -->
> >> > >> : <lst name="defaults">
> >> > >> : <str name="echoParams">all</str>
> >> > >> : <int name="rows">20</int>
> >> > >> : <!-- Query settings -->
> >> > >> : <str name="df">text</str>
> >> > >> : <!-- <str name="df">title</str> -->
> >> > >> : <str name="defType">synonym_edismax</str>>
> >> > >> : <str name="synonyms">true</str>
> >> > >> : <!-- The line below balances out the weighting of exact
> >> matches to
> >> > >> the
> >> > >> : synonym phrase entered by the user
> >> > >> : with the category_weight calculation and the titleQuery
> >> calc.
> >> > >> : These numbers exist in a balance and
> >> > >> : if one is raised or lowered, the others (probably) need
> to
> >> > >> change
> >> > >> : as well. It may be better to go with decimals
> >> > >> : for all of them... .4 instead of 4 and 2 instead of 20
> and
> >> > 2.5
> >> > >> : instead of 25.
> >> > >> : In the end, I'm not sure it really matters, but don't
> >> change
> >> > >> one
> >> > >> : without changing the others
> >> > >> : unless you've tested and are sure you want the results
> >> -->
> >> > >> : <float name="synonyms.originalBoost">1.5</float>
> >> > >> : <float name="synonyms.synonymBoost">1.1</float>
> >> > >> : <str name="mm">75%</str>
> >> > >> : <str name="q.alt">*:*</str>
> >> > >> : <str name="rows">20</str>
> >> > >> : <str name="fq">meta_doc_type:chapterDoc</str>
> >> > >> : <str name="bq">{!synonym_edismax qf='title'
> synonyms='true'
> >> > >> : synonyms.originalBoost='2.5' synonyms.synonymBoost='1.1' bf=''
> >> bq=''
> >> > >> : v=$q}</str>
> >> > >> : <str name="fl">id category_weight title category_ss score
> >> > >> : contentType</str>
> >> > >> : <str name="titleQuery">{!edismax qf='title' bf='' bq=''
> >> > >> v=$q}</str>
> >> > >> : =====================================================
> >> > >> : *<str name="bf">product(field(category_weight),20)</str>*
> >> > >> : =====================================================
> >> > >> : <str name="bf">product(query($titleQuery),4)</str>
> >> > >> : <str name="qf">text contentType^1000</str>
> >> > >> : <str name="wt">python</str>
> >> > >> : <str name="debug">true</str>
> >> > >> : <str name="debug.explain.structured">true</str>
> >> > >> : <str name="indent">true</str>
> >> > >> : <str name="echoParams">all</str>
> >> > >> : </lst>
> >> > >> : </requestHandler>
> >> > >> :
> >> > >> : And here is the debug output for a query. (This was a test for
> >> > >> synonyms,
> >> > >> : which you'll see in the output.) The original query string was,
> of
> >> > >> : course, "μ-heavy
> >> > >> : chain disease"
> >> > >> :
> >> > >> : You'll note that although there is no score in the first doc
> >> explain
> >> > for
> >> > >> : the actual querystring, the highlighted section does get a score
> >> for
> >> > >> : product(double(category_weight)=1.5,const(20))
> >> > >> :
> >> > >> : ... which is the thing that is currently causing all the docs in
> >> the
> >> > >> : collection to "match" even though the querystring is not in any
> of
> >> > them.
> >> > >> :
> >> > >> : "debug":{ "rawquerystring":"\"μ-heavy chain disease\"",
> >> > >> : "querystring":"\"μ-heavy
> >> > >> : chain disease\"", "parsedquery":"(DisjunctionMaxQuery((text:\"μ
> >> heavy
> >> > >> chain
> >> > >> : disease\" | (contentType:\"μ heavy chain disease\")^1000.0))^1.5
> >> > >> : ((+DisjunctionMaxQuery((text:\"mu heavy chain disease\" |
> >> > >> (contentType:\"mu
> >> > >> : heavy chain disease\")^1000.0)))/no_coord^1.1)
> >> > >> : ((+DisjunctionMaxQuery((text:\"μ hcd\" | (contentType:\"μ
> >> > >> : hcd\")^1000.0)))/no_coord^1.1) ((+DisjunctionMaxQuery((text:\"μ
> >> heavy
> >> > >> chain
> >> > >> : disease\" | (contentType:\"μ heavy chain
> >> > disease\")^1000.0)))/no_coord^
> >> > >> 1.1)
> >> > >> : ((+DisjunctionMaxQuery((text:\"μ hcd\" | (contentType:\"μ
> >> > >> : hcd\")^1000.0)))/no_coord^1.1)) ((DisjunctionMaxQuery((title:\"μ
> >> > heavy
> >> > >> : chain disease\"))^2.5 ((+DisjunctionMaxQuery((title:\"mu heavy
> >> chain
> >> > >> : disease\")))/no_coord^1.1) ((+DisjunctionMaxQuery((title:\"μ
> >> > >> : hcd\")))/no_coord^1.1) ((+DisjunctionMaxQuery((title:\"μ heavy
> >> chain
> >> > >> : disease\")))/no_coord^1.1) ((+DisjunctionMaxQuery((title:\"μ
> >> > >> : hcd\")))/no_coord^1.1)))
> >> > >> : FunctionQuery(product(double(category_weight),const(20)))
> >> > >> : FunctionQuery(product(query(+(title:\"μ heavy chain
> >> > >> : disease\"),def=0.0),const(4)))", "parsedquery_toString":"(((tex
> >> t:\"μ
> >> > >> heavy
> >> > >> : chain disease\" | (contentType:\"μ heavy chain
> >> disease\")^1000.0))^1.5
> >> > >> : ((+(text:\"mu heavy chain disease\" | (contentType:\"mu heavy
> chain
> >> > >> : disease\")^1000.0))^1.1) ((+(text:\"μ hcd\" | (contentType:\"μ
> >> > >> : hcd\")^1000.0))^1.1) ((+(text:\"μ heavy chain disease\" |
> >> > >> (contentType:\"μ
> >> > >> : heavy chain disease\")^1000.0))^1.1) ((+(text:\"μ hcd\" |
> >> > >> (contentType:\"μ
> >> > >> : hcd\")^1000.0))^1.1)) ((((title:\"μ heavy chain disease\"))^2.5
> >> > >> : ((+(title:\"mu heavy chain disease\"))^1.1) ((+(title:\"μ
> >> hcd\"))^1.1)
> >> > >> : ((+(title:\"μ heavy chain disease\"))^1.1) ((+(title:\"μ
> >> > hcd\"))^1.1)))
> >> > >> : product(double(category_weight),const(20))
> >> product(query(+(title:\"μ
> >> > >> heavy
> >> > >> : chain disease\"),def=0.0),const(4))", "explain":{ "
> >> > >> : 33d808fe-6ccf-4305-a643-48e94de34d18":{ "match":true,
> >> "value":30.0, "
> >> > >> : description":"sum of:", "details":[{ "match":true, "value":30.0,
> "
> >> > >> : description":"FunctionQuery(product(double(category_weight),
> >> > >> const(20))),
> >> > >> : product of:",
> >> > >> : =====================================================
> >> > >> : *"details":**[{ "match":true, "value":30.0,
> >> > >> : "description":"product(double(category_weight)=1.5,const(20))"},
> >> {*
> >> > >> : =====================================================
> >> > >> :
> >> > >> : "match":true, "value":1.0, "description":"boost"}, {
> "match":true,
> >> > >> "value":
> >> > >> : 1.0, "description":"queryNorm"}]}, {
> >> > >> :
> >> > >>
> >> > >> -Hoss
> >> > >> http://www.lucidworks.com/
> >> > >
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
