Distance sorting with spatial filtering
The new spatial filtering (SOLR-1586) works great and is much faster
than fq={!frange. However, I am having problems sorting by distance.
If I try
GET
'http://localhost:8983/solr/select/?q=*:*&sort=dist(2,latitude,longitude,0,0)+asc'
I get an error:
Error 400 can not sort on unindexed field: dist(2,latitude,longitude,0,0)
I was able to work around this with
GET 'http://localhost:8983/solr/select/?q=*:* AND _val_:"recip(dist(2,
latitude, longitude, 0,0),1,1,1)"&fl=*,score'
But why isn't sorting by functions working? I get this error with any
function I try to sort on.This is a nightly trunk build from Aug 25th.
I see SOLR-1297 was reopened, but that seems to be for edge cases.
Second question: I am using the LatLonType from the Spatial Filtering
wiki, http://wiki.apache.org/solr/SpatialSearch
Are there any distance sorting functions that use this field, or do I
need to have three indexed fields, store_lat_lon, latitude, and
longitude, if I want both filtering and sorting by distance.
Thanks, Scott
Re: Distance sorting with spatial filtering
I get the error on all functions.
GET 'http://localhost:8983/solr/select?q=*:*&sort=sum(1)+asc'
Error 400 can not sort on unindexed field: sum(1)
I tried another nightly build from today, Sep 7th, with the same
results. I attached the schema.xml
Thanks for the help!
Scott
On Wed, Sep 1, 2010 at 18:43, Lance Norskog wrote:
> Post your schema.
>
> On Mon, Aug 30, 2010 at 2:04 PM, Scott K wrote:
>> The new spatial filtering (SOLR-1586) works great and is much faster
>> than fq={!frange. However, I am having problems sorting by distance.
>> If I try
>> GET
>> 'http://localhost:8983/solr/select/?q=*:*&sort=dist(2,latitude,longitude,0,0)+asc'
>> I get an error:
>> Error 400 can not sort on unindexed field: dist(2,latitude,longitude,0,0)
>>
>> I was able to work around this with
>> GET 'http://localhost:8983/solr/select/?q=*:* AND _val_:"recip(dist(2,
>> latitude, longitude, 0,0),1,1,1)"&fl=*,score'
>>
>> But why isn't sorting by functions working? I get this error with any
>> function I try to sort on.This is a nightly trunk build from Aug 25th.
>> I see SOLR-1297 was reopened, but that seems to be for edge cases.
>>
>> Second question: I am using the LatLonType from the Spatial Filtering
>> wiki, http://wiki.apache.org/solr/SpatialSearch
>> Are there any distance sorting functions that use this field, or do I
>> need to have three indexed fields, store_lat_lon, latitude, and
>> longitude, if I want both filtering and sorting by distance.
>>
>> Thanks, Scott
>>
>
>
>
> --
> Lance Norskog
> goks...@gmail.com
>
listing_id
text
Re: Distance sorting with spatial filtering
According to the documentation, sorting by function has been a feature
since Solr 1.5. It seems like a major regression if this no longer
works.
http://wiki.apache.org/solr/FunctionQuery#Sort_By_Function
The _val_ trick does not seem to work if used with a query term,
although I can try some more things to give 0 value to the query term.
On Wed, Sep 8, 2010 at 22:21, Lance Norskog wrote:
> It says that the field "sum(1)" is not indexed. You don't have a field
> called 'sum(1)'. I know there has been a lot of changes in query parsing,
> and sorting by functions may be on the list. But the _val_ trick is the
> older one and, and you noted, still works. The _val_ trick sets the ranking
> value to the output of the function, thus indirectly doing what sort= does.
>
> Lance
>
> Scott K wrote:
>>
>> I get the error on all functions.
>> GET 'http://localhost:8983/solr/select?q=*:*&sort=sum(1)+asc'
>> Error 400 can not sort on unindexed field: sum(1)
>>
>> I tried another nightly build from today, Sep 7th, with the same
>> results. I attached the schema.xml
>>
>> Thanks for the help!
>> Scott
>>
>> On Wed, Sep 1, 2010 at 18:43, Lance Norskog wrote:
>>
>>>
>>> Post your schema.
>>>
>>> On Mon, Aug 30, 2010 at 2:04 PM, Scott K wrote:
>>>
>>>>
>>>> The new spatial filtering (SOLR-1586) works great and is much faster
>>>> than fq={!frange. However, I am having problems sorting by distance.
>>>> If I try
>>>> GET
>>>> 'http://localhost:8983/solr/select/?q=*:*&sort=dist(2,latitude,longitude,0,0)+asc'
>>>> I get an error:
>>>> Error 400 can not sort on unindexed field:
>>>> dist(2,latitude,longitude,0,0)
>>>>
>>>> I was able to work around this with
>>>> GET 'http://localhost:8983/solr/select/?q=*:* AND _val_:"recip(dist(2,
>>>> latitude, longitude, 0,0),1,1,1)"&fl=*,score'
>>>>
>>>> But why isn't sorting by functions working? I get this error with any
>>>> function I try to sort on.This is a nightly trunk build from Aug 25th.
>>>> I see SOLR-1297 was reopened, but that seems to be for edge cases.
>>>>
>>>> Second question: I am using the LatLonType from the Spatial Filtering
>>>> wiki, http://wiki.apache.org/solr/SpatialSearch
>>>> Are there any distance sorting functions that use this field, or do I
>>>> need to have three indexed fields, store_lat_lon, latitude, and
>>>> longitude, if I want both filtering and sorting by distance.
>>>>
>>>> Thanks, Scott
>>>>
>>>>
>>>
>>>
>>> --
>>> Lance Norskog
>>> goks...@gmail.com
>>>
>>>
>
Re: Is there a way to fetch the complete list of data from a particular column in SOLR document?
But how do you know when the document actually makes it to solr, especially if you are using commitWithin and not explicitly calling commit. One solution is to have a status field in the database such as 0 - unindexed 1 - indexing 2 - committed / verified And have a separate process query solr for documents in the indexing state and set them to committed if they are queryable in solr. On Tue, Sep 7, 2010 at 14:26, Geert-Jan Brits wrote: >>Please let me know if there are any other ideas / suggestions to implement > this. > > You're indexing program should really take care of this IMHO. Each time your > indexer inserts a document to Solr, flag the corresponding entity in your > RDBMS, each time you delete, remove the flag. You should implement this as a > transaction to make sure all is still fine in the unlikely event of a crash > midway. > > 2010/9/7 bbarani > >> >> Hi, >> >> I am trying to get complete list of unique document ID and compare it with >> that of back end to make sure that both back end and SOLR documents are in >> sync. >> >> Is there a way to fetch the complete list of data from a particular column >> in SOLR document? >> >> Once I get the list, I can easily compare it against the DB and delete the >> orphan documents.. >> >> Please let me know if there are any other ideas / suggestions to implement >> this. >> >> Thanks, >> Barani >> -- >> View this message in context: >> http://lucene.472066.n3.nabble.com/Is-there-a-way-to-fetch-the-complete-list-of-data-from-a-particular-column-in-SOLR-document-tp1435586p1435586.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >
Re: Distance sorting with spatial filtering
On Thu, Sep 9, 2010 at 21:00, Lance Norskog wrote:
> I just checked out the trunk, and branch 3.x This query is accepted on both,
> but gives no responses:
> http://localhost:8983/solr/select/?q=*:*&sort=dist(2,x_dt,y_dt,0,0)+asc
So you are saying when you add the sort parameter you get no results
back, but do not get the error I am seeing? Should I open a Jira
ticket?
> x_dt and y_dt are wildcard fields with the tdouble type. "tdouble"
> explicitly says it is stored and indexed. Your 'longitude' and 'latitude'
> fields may not be stored?
No, they are stored.
http://localhost:8983/solr/select?q=*:*&rows=1&wt=xml&indent=true
0
9
...
47.6636
-122.3054
> Also, this is accepted on both branches:
> http://localhost:8983/solr/select/?q=*:*&sort=sum(1)+asc
>
> The documentation for sum() does not mention single-argument calls.
This also fails
http://localhost:8983/solr/select/?q=*:*&sort=sum(1,2)+asc
http://localhost:8983/solr/select/?q=*:*&sort=sum(latitude,longitude)+asc
> Scott K wrote:
>>
>> According to the documentation, sorting by function has been a feature
>> since Solr 1.5. It seems like a major regression if this no longer
>> works.
>> http://wiki.apache.org/solr/FunctionQuery#Sort_By_Function
>>
>> The _val_ trick does not seem to work if used with a query term,
>> although I can try some more things to give 0 value to the query term.
>>
>> On Wed, Sep 8, 2010 at 22:21, Lance Norskog wrote:
>>
>>>
>>> It says that the field "sum(1)" is not indexed. You don't have a field
>>> called 'sum(1)'. I know there has been a lot of changes in query parsing,
>>> and sorting by functions may be on the list. But the _val_ trick is the
>>> older one and, and you noted, still works. The _val_ trick sets the
>>> ranking
>>> value to the output of the function, thus indirectly doing what sort=
>>> does.
>>>
>>> Lance
>>>
>>> Scott K wrote:
>>>
>>>>
>>>> I get the error on all functions.
>>>> GET 'http://localhost:8983/solr/select?q=*:*&sort=sum(1)+asc'
>>>> Error 400 can not sort on unindexed field: sum(1)
>>>>
>>>> I tried another nightly build from today, Sep 7th, with the same
>>>> results. I attached the schema.xml
>>>>
>>>> Thanks for the help!
>>>> Scott
>>>>
>>>> On Wed, Sep 1, 2010 at 18:43, Lance Norskog wrote:
>>>>
>>>>
>>>>>
>>>>> Post your schema.
>>>>>
>>>>> On Mon, Aug 30, 2010 at 2:04 PM, Scott K wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> The new spatial filtering (SOLR-1586) works great and is much faster
>>>>>> than fq={!frange. However, I am having problems sorting by distance.
>>>>>> If I try
>>>>>> GET
>>>>>>
>>>>>> 'http://localhost:8983/solr/select/?q=*:*&sort=dist(2,latitude,longitude,0,0)+asc'
>>>>>> I get an error:
>>>>>> Error 400 can not sort on unindexed field:
>>>>>> dist(2,latitude,longitude,0,0)
>>>>>>
>>>>>> I was able to work around this with
>>>>>> GET 'http://localhost:8983/solr/select/?q=*:* AND _val_:"recip(dist(2,
>>>>>> latitude, longitude, 0,0),1,1,1)"&fl=*,score'
>>>>>>
>>>>>> But why isn't sorting by functions working? I get this error with any
>>>>>> function I try to sort on.This is a nightly trunk build from Aug 25th.
>>>>>> I see SOLR-1297 was reopened, but that seems to be for edge cases.
>>>>>>
>>>>>> Second question: I am using the LatLonType from the Spatial Filtering
>>>>>> wiki, http://wiki.apache.org/solr/SpatialSearch
>>>>>> Are there any distance sorting functions that use this field, or do I
>>>>>> need to have three indexed fields, store_lat_lon, latitude, and
>>>>>> longitude, if I want both filtering and sorting by distance.
>>>>>>
>>>>>> Thanks, Scott
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Lance Norskog
>>>>> goks...@gmail.com
>>>>>
>>>>>
>>>>>
>>>
>>>
>
Re: Distance sorting with spatial filtering
I tracked down the problem and found a workaround. If there is a
wildcard entry in schema.xml such as the following.
then sort by function fails and returns Error 400 can not sort on
unindexed field:
Removing the name="*" entry from schema.xml is a workaround. I noted
this in the Solr-1297 JIRA entry.
Scott
On Fri, Sep 10, 2010 at 01:40, Lance Norskog wrote:
> Since no one has jumped in to give the right syntax- yeah, it's a bug.
> Please file a JIRA.
>
> On Thu, Sep 9, 2010 at 9:44 PM, Scott K wrote:
>> On Thu, Sep 9, 2010 at 21:00, Lance Norskog wrote:
>>> I just checked out the trunk, and branch 3.x This query is accepted on both,
>>> but gives no responses:
>>> http://localhost:8983/solr/select/?q=*:*&sort=dist(2,x_dt,y_dt,0,0)+asc
>>
>> So you are saying when you add the sort parameter you get no results
>> back, but do not get the error I am seeing? Should I open a Jira
>> ticket?
>>
>>> x_dt and y_dt are wildcard fields with the tdouble type. "tdouble"
>>> explicitly says it is stored and indexed. Your 'longitude' and 'latitude'
>>> fields may not be stored?
>>
>> No, they are stored.
>> http://localhost:8983/solr/select?q=*:*&rows=1&wt=xml&indent=true
>>
>>
>>
>> 0
>> 9
>>
>>
>>
>> ...
>> 47.6636
>> -122.3054
>>
>>
>>> Also, this is accepted on both branches:
>>> http://localhost:8983/solr/select/?q=*:*&sort=sum(1)+asc
>>>
>>> The documentation for sum() does not mention single-argument calls.
>>
>> This also fails
>> http://localhost:8983/solr/select/?q=*:*&sort=sum(1,2)+asc
>> http://localhost:8983/solr/select/?q=*:*&sort=sum(latitude,longitude)+asc
>>
>>
>>> Scott K wrote:
>>>>
>>>> According to the documentation, sorting by function has been a feature
>>>> since Solr 1.5. It seems like a major regression if this no longer
>>>> works.
>>>> http://wiki.apache.org/solr/FunctionQuery#Sort_By_Function
>>>>
>>>> The _val_ trick does not seem to work if used with a query term,
>>>> although I can try some more things to give 0 value to the query term.
>>>>
>>>> On Wed, Sep 8, 2010 at 22:21, Lance Norskog wrote:
>>>>
>>>>>
>>>>> It says that the field "sum(1)" is not indexed. You don't have a field
>>>>> called 'sum(1)'. I know there has been a lot of changes in query parsing,
>>>>> and sorting by functions may be on the list. But the _val_ trick is the
>>>>> older one and, and you noted, still works. The _val_ trick sets the
>>>>> ranking
>>>>> value to the output of the function, thus indirectly doing what sort=
>>>>> does.
>>>>>
>>>>> Lance
>>>>>
>>>>> Scott K wrote:
>>>>>
>>>>>>
>>>>>> I get the error on all functions.
>>>>>> GET 'http://localhost:8983/solr/select?q=*:*&sort=sum(1)+asc'
>>>>>> Error 400 can not sort on unindexed field: sum(1)
>>>>>>
>>>>>> I tried another nightly build from today, Sep 7th, with the same
>>>>>> results. I attached the schema.xml
>>>>>>
>>>>>> Thanks for the help!
>>>>>> Scott
>>>>>>
>>>>>> On Wed, Sep 1, 2010 at 18:43, Lance Norskog wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Post your schema.
>>>>>>>
>>>>>>> On Mon, Aug 30, 2010 at 2:04 PM, Scott K wrote:
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> The new spatial filtering (SOLR-1586) works great and is much faster
>>>>>>>> than fq={!frange. However, I am having problems sorting by distance.
>>>>>>>> If I try
>>>>>>>> GET
>>>>>>>>
>>>>>>>> 'http://localhost:8983/solr/select/?q=*:*&sort=dist(2,latitude,longitude,0,0)+asc'
>>>>>>>> I get an error:
>>>>>>>> Error 400 can not sort on unindexed field:
>>>>>>>> dist(2,latitude,longitude,0,0)
>>>>>>>>
>>>>>>>> I was able to work around this with
>>>>>>>> GET 'http://localhost:8983/solr/select/?q=*:* AND _val_:"recip(dist(2,
>>>>>>>> latitude, longitude, 0,0),1,1,1)"&fl=*,score'
>>>>>>>>
>>>>>>>> But why isn't sorting by functions working? I get this error with any
>>>>>>>> function I try to sort on.This is a nightly trunk build from Aug 25th.
>>>>>>>> I see SOLR-1297 was reopened, but that seems to be for edge cases.
>>>>>>>>
>>>>>>>> Second question: I am using the LatLonType from the Spatial Filtering
>>>>>>>> wiki, http://wiki.apache.org/solr/SpatialSearch
>>>>>>>> Are there any distance sorting functions that use this field, or do I
>>>>>>>> need to have three indexed fields, store_lat_lon, latitude, and
>>>>>>>> longitude, if I want both filtering and sorting by distance.
>>>>>>>>
>>>>>>>> Thanks, Scott
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Lance Norskog
>>>>>>> goks...@gmail.com
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>
>
>
>
> --
> Lance Norskog
> goks...@gmail.com
>
How do I get the solr error response as XML instead of HTML
solr errors come back as HTML instead of XM or JSON Is it possible to get the response to come back as XML or JSON, or at least something I could show to an end user? Is there a way to tell solr to ignore unparseable terms and still return a result, ideally with a warning so the end user doesn't get an error page. GET 'http://localhost:8983/solr/select/?q=term+OR&wt=xml' Error 400 org.apache.lucene.queryParser.ParseException: Cannot parse 'term OR': Encountered "" at line 1, column 7. Was expecting one of: ... "+" ... "-" ... "(" ... "*" ... ... ... ... ... ... "[" ... "{" ... ... ... "*" ... HTTP ERROR 400 Problem accessing /solr/select/. Reason: org.apache.lucene.queryParser.ParseException: Cannot parse 'term OR': Encountered " " at line 1, column 7. Was expecting one of: ... "+" ... "-" ... "(" ... "*" ... ... ... ... ... ... "[" ... "{" ... ... ... "*" ... Powered by Jetty://
Re: NOT keyword - doesn't work with dismax?
I couldn't even get the bq= to work with negated queries, although
with edismax, negated queries work with just q=-term
Works:
/solr/select?qt=edismax&q=-red
Here is the failed attempt with dismax
/solr/select?qt=dismax&rows=1&indent=true&q=-red&bq=*:*^0.001&echoParams=all&debugQuery=true
{
"responseHeader":{
"status":0,
"QTime":20,
"params":{
"mm":"2<-1 5<-2 6<90%",
"pf":"title^10.0 sbody^2.0",
"echoParams":"all",
"tie":"0.01",
"qf":"title^10.0 sbody^2.0 tags^1.0 text^1.0",
"q.alt":"*:*",
"hl.fl":"body",
"wt":"json",
"ps":"100",
"defType":"dismax",
"bq":"*:*^0.001",
"echoParams":"all",
"debugQuery":"true",
"indent":"true",
"q":"-red",
"qt":"dismax",
"rows":"1"}},
"response":{"numFound":0,"start":0,"docs":[]
},
"debug":{
"rawquerystring":"-red",
"querystring":"-red",
"parsedquery":"+(-DisjunctionMaxQuery((tags:red | text:red |
title:red^10.0 | sbody:red^2.0)~0.01))
DisjunctionMaxQuery((title:red^10.0 | sbody:red^2.0)~0.01)
MatchAllDocsQuery(*:*^0.0010)",
"parsedquery_toString":"+(-(tags:red | text:red | title:red^10.0 |
sbody:red^2.0)~0.01) (title:red^10.0 | sbody:red^2.0)~0.01
*:*^0.0010",
"explain":{},
"QParser":"DisMaxQParser",
"altquerystring":null,
"boost_queries":["*:*^0.001"],
"parsed_boost_queries":["MatchAllDocsQuery(*:*^0.0010)"],
"boostfuncs":null,
"timing":{
"time":20.0,
"prepare":{
"time":19.0,
"org.apache.solr.handler.component.QueryComponent":{
"time":19.0},
"org.apache.solr.handler.component.FacetComponent":{
"time":0.0},
"org.apache.solr.handler.component.MoreLikeThisComponent":{
"time":0.0},
"org.apache.solr.handler.component.HighlightComponent":{
"time":0.0},
"org.apache.solr.handler.component.StatsComponent":{
"time":0.0},
"org.apache.solr.handler.component.DebugComponent":{
"time":0.0}},
"process":{
"time":1.0,
"org.apache.solr.handler.component.QueryComponent":{
"time":0.0},
"org.apache.solr.handler.component.FacetComponent":{
"time":0.0},
"org.apache.solr.handler.component.MoreLikeThisComponent":{
"time":0.0},
"org.apache.solr.handler.component.HighlightComponent":{
"time":0.0},
"org.apache.solr.handler.component.StatsComponent":{
"time":0.0},
"org.apache.solr.handler.component.DebugComponent":{
"time":1.0}
On Wed, Apr 28, 2010 at 23:35, Chris Hostetter wrote:
>
> : Ah, dismax doesn't support top-level NOT query.
>
> Hmm, yeah i don' think support for purely negated queries was ever added
> to dismax.
>
> I'm pretty sure that as a workarround you can use add
> something like...
> bq=*:*^0.001
> ...to your query. based on the dismax structure, that should allow purely
> negative queries to work.
>
>
>
> -Hoss
>
>
Re: Spatial search in Solr 1.5
I just upgraded to a later version of the trunk and noticed my geofilter queries stopped working, apparently because the sfilt function was renamed to geofilt. I realize trunk is not stable, but other than looking at every change, is there an easy way to find changes that are not backward compatible so developers know what they need to update when upgrading? Thanks, Scott On Tue, Oct 12, 2010 at 17:42, Yonik Seeley wrote: > On Tue, Oct 12, 2010 at 8:07 PM, PeterKerk wrote: >> >> Ok, so does this actually say: >> for now you have to do calculations based on bounding box instead of great >> circle? > > I tried to make the documentation a little simpler... there's > - geofilt... filters within a radius of "d" km (i.e. "great circle > distance") > - bbox... filters using a bounding box > - geodist... function query that yields the distance (again, "great > circle distance") > > If you point out the part to the docs you found confusing, I can try > and improve it. > Did you try and step through the quick start? Those links actually work! > >> And the fact that on top of the page it says "Solr4.0", does that imply I >> cant use this right now? Or where could I find the latest trunk for this? > > The wiki says "If you haven't already, get a recent nightly build of > Solr4.0"... > and links to the Solr4.0 page, which points to > http://wiki.apache.org/solr/FrontPage#solr_development > for nightly builds. > > -Yonik > > http://www.lucidimagination.com >
sort by price puts unknown prices first
When I sort by price ascending, documents with no price are listed first. I would like them listed last. I tried adding the sortMissingLast flag, even though it says it is only for strings, but it did not help. Why doesn't sortMissingLast work on non-strings? This seems like a very common issue, but I couldn't find any solutions when I searched this group and on google. The map function almost works, but if I use this, then prices of 0 are treated as null, which is not what I want. sort=map(price,0,0,99)+asc schema.xml: Thanks, Scott
Re: sort by price puts unknown prices first
On Wed, Mar 2, 2011 at 12:21, Chris Hostetter wrote: > historicly it has been because of a fundemental limitation in how the > Lucene FieldCache has historicly worked where the array backed FieldCaches > use the default numeric value (ie: 0) when docs have no value (but in the > case of Strings, the default is "null" which is easy to test for) > > i am 99.99% certain this has changed on the trunk, so all of the > Trie*Fields should support the sortMissing* options in 4.x I am running a 4.x build and just tried the most recent nightly build, apache-solr-4.0-2011-03-02_08-06-07.tgz, and am still seeing this issue. Other than creating a new indexed field where I manually map no value to a high number, is there a way to sort on a function query that puts undefined values to the end? Is there no way to use map to change undefined values? Thanks, Scott
