Query regarding LTR plugin in solr
Hi,
I'm working on ltr feature in solr. I have a feature like :
''' {
"store" : "my_feature_store",
"name" : "in_aggregated_terms",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : { "q" : "{!func}scale(query({!payload_score
f=aggregated_terms func=max v=${query}}),0,100)" }
} '''
Here the scaling function is taking a lot more time than expected.
Is there a way I could implement a customized class or any other way by
which I can reduce this time.
So basically I just want to scale the value which looks at the whole result
set instead of just the current document. Can I have/implement something
during normalization??
Thanks in advance
Regards,
Prateek
Query regarding LTR plugin in solr
Hi,
I'm working on ltr feature in solr. I have a feature like :
''' {
"store" : "my_feature_store",
"name" : "in_aggregated_terms",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : { "q" : "{!func}scale(query({!payload_score f=aggregated_terms
func=max v=${query}}),0,100)" }
} '''
Here the scaling function is taking a lot more time than expected.
Is there a way I could implement a customized class or any other way by which I
can reduce this time.
So basically I just want to scale the value which looks at the whole result set
instead of just the current document. Can I have/implement something during
normalization??
Thanks in advance
Regards,
Prateek
Regarding LTR feature
Hi all,
I'm new to solr ltr and stuck on this problem for a while.
I wanted to ask why the documents on which the ltr feature score is
calculated doesn't filter out the documents even if we provide the fq
filter in the url like:
&q=juice&rq={!ltr%20model=my_feature_model%20efi.query=$q%
20reRankDocs=300%20efi.store=1}&fq=parent_store_3630_i:%201
Here the score calculation for features should only use the documents
returned from these fq parameter but it's not really the case. Is it a bug
or something.
Thanks in advance.
Regards,
Prateek
Re: Regarding LTR feature
Hi Alessandro,
Thanks for responding.
Let me take a step back and tell you the problem I have been facing with
this.So one of the features in my LTR model is:
{
"store" : "my_feature_store",
"name" : "in_aggregated_terms",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : { "q" : "{!func}scale(query({!payload_score
f=aggregated_terms func=max v=${query}}),0,100)" }
}
so now with this feature if i apply FQ in solr it will scale the
values for all the documents irrespective of the FQ filter.
But if I change the feature to something like this:
{
"store" : "my_feature_store",
"name" : "in_aggregated_terms",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : { "q" : "{!func}scale(query({!field f=aggregated_terms
v=${query}}),0,100)" }
}
Then the it scales properly with FQ aswell.
And about that verification I simply check the results returned like
in Case 1 after applying the FQ filter that feature score doesn't
scale to its maximum value of 100 which i think is because of the fact
that it scales over all the documents and returns only the subset with
the FQ filter applied.
Alternatively is their any way I can scale these value during
normalization time with a customized class which iterates over all the
re-ranked documents only.
Thanks a lot in advance.
Looking forward to hearing back from you soon.
Regards,
Prateek
Re: Regarding LTR feature
Hi Alessandro,
Thanks for responding.
Let me take a step back and tell you the problem I have been facing with
this.So one of the features in my LTR model is:
{
"store" : "my_feature_store",
"name" : "in_aggregated_terms",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : { "q" : "{!func}scale(query({!payload_score
f=aggregated_terms func=max v=${query}}),0,100)" }
}
so now with this feature if i apply FQ in solr it will scale the
values for all the documents irrespective of the FQ filter.
But if I change the feature to something like this:
{
"store" : "my_feature_store",
"name" : "in_aggregated_terms",
"class" : "org.apache.solr.ltr.feature.SolrFeature",
"params" : { "q" : "{!func}scale(query({!field f=aggregated_terms
v=${query}}),0,100)" }
}
Then the it scales properly with FQ aswell.
And about that verification I simply check the results returned like
in Case 1 after applying the FQ filter that feature score doesn't
scale to its maximum value of 100 which i think is because of the fact
that it scales over all the documents and returns only the subset with
the FQ filter applied.
Alternatively is their any way I can scale these value during
normalization time with a customized class which iterates over all the
re-ranked documents only.
Thanks a lot in advance.
Looking forward to hearing back from you soon.
Regards,
Prateek
Re: Regarding LTR feature
Thanks again Alessandro I tried with the feature and the Minmax normalizer you told.But then there is a slight problem with the params in normalization. I don't really know the range(Min, Max) of values the payload_score outputs and they are different for different queries. I even tried looking at the source code to see if there is a way I can override a class so that it iterates over all the re-ranked documents and calculate Max and min there itself and pass it to MinMax normalizer class but it seems it's not possible. Your help will be really appreciated. Thanks Regards, Prateek
Re: Regarding LTR feature
Thanks again Alessandro
I tried with the feature and the Minmax normalizer you told.But then there is a
slight problem with the params in normalization. I don't really know the
range(Min, Max) of values the payload_score outputs and they are different for
different queries.
I even tried looking at the source code to see if there is a way I can override
a class so that it iterates over all the re-ranked documents and calculate Max
and min there itself and pass it to MinMax normalizer class but it seems it's
not possible.
Your help will really appreciated.
Thanks
Regards,
Prateek
On 2018/05/03 14:00:00, Alessandro Benedetti wrote:
> Mmmm, first of all, you know that each Solr feature is calculated per
> document right ?
> So you want to calculate the payload score for the document you are
> re-ranking, based on the query ( your External Feature Information) and
> normalize across the different documents?
>
> I would go with this feature and use the normalization LTR functionality :
>
> {
> "store" : "my_feature_store",
> "name" : "in_aggregated_terms",
> "class" : "org.apache.solr.ltr.feature.SolrFeature",
> "params" : { "q" : "{!payload_score
> f=aggregated_terms func=max v=${query}}" }
> }
>
> Then in the model you specify something like :
>
> "name" : "myModelName",
>"features" : [
>{
> "name" : "isBook"
>},
> ...
>{
> "name" : "in_aggregated_terms",
> "norm": {
> "class" : "org.apache.solr.ltr.norm.MinMaxNormalizer",
> "params" : { "min":"x", "max":"y" }
> }
>},
>}
>
> Give it a try, let me know
>
>
>
>
> -
> ---
> Alessandro Benedetti
> Search Consultant, R&D Software Engineer, Director
> Sease Ltd. - www.sease.io
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Re: Regarding LTR feature
Hi Alessandro, You're right it doesn't have to be that accurate to the query time but our requirement is having a more solid control over our outputs from Solr like if we have 4 features then we can adjust the weights giving something like (40,20,20,20) to each feature such that the sum total of features for a document is 100 this is only possible if we could scale the feature outputs between 0-1. Secondly, I also have a doubt regarding the scaling function like why it is not considering only the documents filtered out by the FQ filter and considering all the documents which match the query. Thaks a lot in advance. Looking forward to hearing back from you soon. Regards, Prateek On 2018/05/04 10:26:55, Alessandro Benedetti wrote: > Hi Preteek, > I would assume you have that feature at training time as well, can't you use > the training set to estabilish the parameters for the normalizer at query > time ? > > In the end being a normalization, doesn't have to be that accurate to the > query time state, but it must reflect the relations the model learnt from > the training set. > Let me know ! > > > > - > --- > Alessandro Benedetti > Search Consultant, R&D Software Engineer, Director > Sease Ltd. - www.sease.io > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html >
Sorting on pseudo field(The one which is added during doctransformer)
Hi all, I wanted to ask is it possible to sort on the field which is added during DocTransformer. I'm referring to something like this: (https://mariofebbraio.files.wordpress.com/2014/10/doctransformer.png) here the price is added using doctransformer so is it possible to sort on this field if I add that field dynamically to the every returned document. If Yes, Can you please guide me like how to go about it. Can anyone also tell me like at what stage sorting is performed after doctransformer or after the requestHandler?? Sorry if you find these queries silly but I'm really confused where all this happens. Thanks in advance. Regards, Prateek
Re: Sorting on pseudo field(The one which is added during doctransformer)
Hi Mikhail, > You can either sort by function that needs to turn the logic into value > source parser. But like my requirement for this was to add a field dynamically from cache or external source to the returned documents from the solr and perform sorting in the solr itself if required otherwise use the score to sort. So how would you advise to go about this?? And how to go about your way "to turn the logic into value source parser" like how to do this for this case?? > If you need to toss just result page, check rerank. I don't want to use it to rank the relevancy of results. Thanks for the response. Regards, Prateek
Re: Sorting on pseudo field(The one which is added during doctransformer)
Hi Mikhail, I think you forgot to link the reference. Thanks Regards, Prateek On 2018/05/17 13:18:22, Mikhail Khludnev wrote: > Here is the reference I've found so far. > > On Thu, May 17, 2018 at 12:26 PM, prateek.agar...@bigbasket.com < > prateek.agar...@bigbasket.com> wrote: > > > > > Hi Mikhail, > > > > > You can either sort by function that needs to turn the logic into value > > > source parser. > > > > But like my requirement for this was to add a field dynamically from cache > > or external source to the returned documents from the solr and perform > > sorting in the solr itself if required otherwise use the score to sort. > > So how would you advise to go about this?? > > > > And how to go about your way "to turn the logic into value source parser" > > like how to do this for this case?? > > > > > > > If you need to toss just result page, check rerank. > > > > I don't want to use it to rank the relevancy of results. > > > > Thanks for the response. > > > > > > > > Regards, > > Prateek > > > > > > -- > Sincerely yours > Mikhail Khludnev >
Re: Sorting on pseudo field(The one which is added during doctransformer)
Thanks, Mikhail On 2018/05/18 11:33:09, Mikhail Khludnev wrote: > Right > https://wiki.apache.org/solr/SolrPlugins#ValueSourceParser > > On Fri, May 18, 2018 at 8:04 AM, prateek.agar...@bigbasket.com < > prateek.agar...@bigbasket.com> wrote: > > > Hi Mikhail, > > > > I think you forgot to link the reference. > > > > Thanks > > > > > > Regards, > > Prateek > > > > On 2018/05/17 13:18:22, Mikhail Khludnev wrote: > > > Here is the reference I've found so far. > > > > > > On Thu, May 17, 2018 at 12:26 PM, prateek.agar...@bigbasket.com < > > > prateek.agar...@bigbasket.com> wrote: > > > > > > > > > > > Hi Mikhail, > > > > > > > > > You can either sort by function that needs to turn the logic into > > value > > > > > source parser. > > > > > > > > But like my requirement for this was to add a field dynamically from > > cache > > > > or external source to the returned documents from the solr and perform > > > > sorting in the solr itself if required otherwise use the score to sort. > > > > So how would you advise to go about this?? > > > > > > > > And how to go about your way "to turn the logic into value source > > parser" > > > > like how to do this for this case?? > > > > > > > > > > > > > If you need to toss just result page, check rerank. > > > > > > > > I don't want to use it to rank the relevancy of results. > > > > > > > > Thanks for the response. > > > > > > > > > > > > > > > > Regards, > > > > Prateek > > > > > > > > > > > > > > > > -- > > > Sincerely yours > > > Mikhail Khludnev > > > > > > > > > -- > Sincerely yours > Mikhail Khludnev >
