custom sorting

[Sandeep Shetty]

> Hi Guys,
> 
> this question as been asked before but i was unable to find an answer
> thats good for me, so hope you guys can help again
> i am working on a website where we need to sort the results by distance
> from the location entered by the user. I have indexed the lat and long
> info for each record in solr and also i can get the lat and long of the
> location input by the user.
> Previously we were using lucene to do this. by using the
> SortComparatorSource we could sort the documents returned by distance
> nicely. we are now switching over to lucene because of the features it
> provides, however i am not able to see a way to do this in Solr. 
> 
> If someone can point me in the right direction i would be very grateful!
> 
> Thanks in advance,
> Sandeep

This email is confidential and may also be privileged. If you are not the 
intended recipient please notify us immediately by telephoning +44 (0)20 7452 
5300 or email [EMAIL PROTECTED] You should not copy it or use it for any 
purpose nor disclose its contents to any other person. Touch Local cannot 
accept liability for statements made which are clearly the sender's own and are 
not made on behalf of the firm.

Touch Local Limited
Registered Number: 2885607
VAT Number: GB896112114
Cardinal Tower, 12 Farringdon Road, London EC1M 3NN
+44 (0)20 7452 5300

RE: custom sorting

[Sandeep Shetty]

i have been testing locallucene with our data for the last couple of days.
one issue i faced with it is during when using geo sorting is that it seems
to eat up all the memory, however big and become progressively slower,
finally after several requests (10 or so in my case) it throws up a
java.lang.OutOfMemoryError: Java heap space error.

is there a way to get around this?

-Original Message-
From: Jon Pierce [mailto:[EMAIL PROTECTED]
Sent: 28 September 2007 15:48
To: solr-user@lucene.apache.org
Subject: Re: custom sorting


Is the machinery in place to do this now (hook up a function query to
be used in sorting)?

I'm trying to figure out what's the best way to do a distance sort:
custom comparator or function query.

Using a custom comparator seems straightforward and reusable across
both the standard and dismax handlers.  But it also seems most likely
to impact performance (or at least require the most work/knowledge to
get right by minimizing calculations, caching, watching out for memory
leaks, etc.).  (Speaking of which, could anyone with more Lucene/Solr
experience than I comment on the performance characteristics of the
locallucene implementation mentioned on the list recently?  I've taken
a first look and it seems reasonable to me.)

Using a function query, as Yonik suggests above, is another approach.
But to get a true sort, you have to boost the original query to zero?
How does this impact the results returned by the original query?  Will
the requirements (and boosts) of the original (now nested) query
remain intact, only sorted by the function?  Also, is there any way to
do this with the dismax handler?

Thanks,
- Jon

On 9/27/07, Yonik Seeley <[EMAIL PROTECTED]> wrote:
> On 9/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
> > Using something like this, how would the custom SortComparatorSource
> > get a parameter from the request to use in sorting calculations?
>
> perhaps hook in via function query:
>   dist(10.4,20.2,geoloc)
>
> And either manipulate the score with that and sort by score,
>
> q=+(foo bar)^0 dist(10.4,20.2,geoloc)
> sort=score asc
>
> or extend solr's sorting mechanisms to allow specifying a function to sort
by.
>
> sort="dist(10.4,20.2,geoloc) asc"
>
> -Yonik
>

This email is confidential and may also be privileged. If you are not the 
intended recipient please notify us immediately by telephoning +44 (0)20 7452 
5300 or email [EMAIL PROTECTED] You should not copy it or use it for any 
purpose nor disclose its contents to any other person. Touch Local cannot 
accept liability for statements made which are clearly the sender's own and are 
not made on behalf of the firm.

Touch Local Limited
Registered Number: 2885607
VAT Number: GB896112114
Cardinal Tower, 12 Farringdon Road, London EC1M 3NN
+44 (0)20 7452 5300

RE: locallucene former custom-sort thread

[Sandeep Shetty]

Hi, i'm using local lucene, downloaded the latest zip file
solr-example_s1.3_ls0.2.tgz

is there a newer version available? 

Thanks!
Sandeep

-Original Message-
From: patrick o'leary [mailto:[EMAIL PROTECTED]
Sent: 28 September 2007 16:08
To: solr-user@lucene.apache.org
Subject: locallucene former custom-sort thread


Changing thread name;

Are you using local lucene or local solr, and which version?


P

[EMAIL PROTECTED] wrote:
> i have been testing locallucene with our data for the last couple of days.
> one issue i faced with it is during when using geo sorting is that it
seems
> to eat up all the memory, however big and become progressively slower,
> finally after several requests (10 or so in my case) it throws up a
> java.lang.OutOfMemoryError: Java heap space error.
>
> is there a way to get around this?
>
> -Original Message-
> From: Jon Pierce [mailto:[EMAIL PROTECTED]
> Sent: 28 September 2007 15:48
> To: solr-user@lucene.apache.org
> Subject: Re: custom sorting
>
>
> Is the machinery in place to do this now (hook up a function query to
> be used in sorting)?
>
> I'm trying to figure out what's the best way to do a distance sort:
> custom comparator or function query.
>
> Using a custom comparator seems straightforward and reusable across
> both the standard and dismax handlers.  But it also seems most likely
> to impact performance (or at least require the most work/knowledge to
> get right by minimizing calculations, caching, watching out for memory
> leaks, etc.).  (Speaking of which, could anyone with more Lucene/Solr
> experience than I comment on the performance characteristics of the
> locallucene implementation mentioned on the list recently?  I've taken
> a first look and it seems reasonable to me.)
>
> Using a function query, as Yonik suggests above, is another approach.
> But to get a true sort, you have to boost the original query to zero?
> How does this impact the results returned by the original query?  Will
> the requirements (and boosts) of the original (now nested) query
> remain intact, only sorted by the function?  Also, is there any way to
> do this with the dismax handler?
>
> Thanks,
> - Jon
>
> On 9/27/07, Yonik Seeley <[EMAIL PROTECTED]> wrote:
>   
>> > On 9/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
>> 
>>> > > Using something like this, how would the custom SortComparatorSource
>>> > > get a parameter from the request to use in sorting calculations?
>>>   
>> >
>> > perhaps hook in via function query:
>> >   dist(10.4,20.2,geoloc)
>> >
>> > And either manipulate the score with that and sort by score,
>> >
>> > q=+(foo bar)0 dist(10.4,20.2,geoloc)
>> > sort=score asc
>> >
>> > or extend solr's sorting mechanisms to allow specifying a function to
sort
>> 
> by.
>   
>> >
>> > sort="dist(10.4,20.2,geoloc) asc"
>> >
>> > -Yonik
>> >
>> 
>
> This email is confidential and may also be privileged. If you are not the
intended recipient please notify us immediately by telephoning +44 (0)20
7452 5300 or email [EMAIL PROTECTED] You should not copy it or use
it for any purpose nor disclose its contents to any other person. Touch
Local cannot accept liability for statements made which are clearly the
sender's own and are not made on behalf of the firm.
>
> Touch Local Limited
> Registered Number: 2885607
> VAT Number: GB896112114
> Cardinal Tower, 12 Farringdon Road, London EC1M 3NN
> +44 (0)20 7452 5300
>
>   

-- 

Patrick O'Leary

AOL Local Search Technologies
Phone: + 1 703 265 8763

You see, wire telegraph is a kind of a very, very long cat. You pull his
tail in New York and his head is meowing in Los Angeles.
 Do you understand this? 
And radio operates exactly the same way: you send signals here, they receive
them there. The only difference is that there is no cat.
  - Albert Einstein

View Patrick O Leary's LinkedIn profileView Patrick O Leary's profile

RE: locallucene former custom-sort thread

[Sandeep Shetty]

also probably a point to consider, the index has about 2.9 million records
in total

-Original Message-
From: Sandeep Shetty 
Sent: 28 September 2007 17:15
To: 'solr-user@lucene.apache.org'
Subject: RE: locallucene former custom-sort thread


yes i was thinking abt the same. 

i was searching for a radius of 25 miles. we get about 2500 results back for
the search. it seems like its storing all those geo results in cache and it
keeps on adding to it each time a geo request is made...

thanks for looking into it! 

Sandeep

-Original Message-
From: patrick o'leary [mailto:[EMAIL PROTECTED]
Sent: 28 September 2007 17:02
To: solr-user@lucene.apache.org
Subject: Re: locallucene former custom-sort thread


That's the latest. I was experimenting with caching, which might be the
problem.
I'll have a look, could you give me an idea of how large the radius was
and how many results were coming back.

Thanks
P

Sandeep Shetty wrote:
> Hi, i'm using local lucene, downloaded the latest zip file
> solr-example_s1.3_ls0.2.tgz
>
> is there a newer version available? 
>
> Thanks!
> Sandeep
>
> -Original Message-
> From: patrick o'leary [mailto:[EMAIL PROTECTED]
> Sent: 28 September 2007 16:08
> To: solr-user@lucene.apache.org
> Subject: locallucene former custom-sort thread
>
>
> Changing thread name;
>
> Are you using local lucene or local solr, and which version?
>
>
> P
>
> [EMAIL PROTECTED] wrote:
>   
>> i have been testing locallucene with our data for the last couple of
days.
>> one issue i faced with it is during when using geo sorting is that it
>> 
> seems
>   
>> to eat up all the memory, however big and become progressively slower,
>> finally after several requests (10 or so in my case) it throws up a
>> java.lang.OutOfMemoryError: Java heap space error.
>>
>> is there a way to get around this?
>>
>> -Original Message-
>> From: Jon Pierce [mailto:[EMAIL PROTECTED]
>> Sent: 28 September 2007 15:48
>> To: solr-user@lucene.apache.org
>> Subject: Re: custom sorting
>>
>>
>> Is the machinery in place to do this now (hook up a function query to
>> be used in sorting)?
>>
>> I'm trying to figure out what's the best way to do a distance sort:
>> custom comparator or function query.
>>
>> Using a custom comparator seems straightforward and reusable across
>> both the standard and dismax handlers.  But it also seems most likely
>> to impact performance (or at least require the most work/knowledge to
>> get right by minimizing calculations, caching, watching out for memory
>> leaks, etc.).  (Speaking of which, could anyone with more Lucene/Solr
>> experience than I comment on the performance characteristics of the
>> locallucene implementation mentioned on the list recently?  I've taken
>> a first look and it seems reasonable to me.)
>>
>> Using a function query, as Yonik suggests above, is another approach.
>> But to get a true sort, you have to boost the original query to zero?
>> How does this impact the results returned by the original query?  Will
>> the requirements (and boosts) of the original (now nested) query
>> remain intact, only sorted by the function?  Also, is there any way to
>> do this with the dismax handler?
>>
>> Thanks,
>> - Jon
>>
>> On 9/27/07, Yonik Seeley <[EMAIL PROTECTED]> wrote:
>>   
>> 
>>>> On 9/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
>>>> 
>>> 
>>>   
>>>>>> Using something like this, how would the custom SortComparatorSource
>>>>>> get a parameter from the request to use in sorting calculations?
>>>>>> 
>>>>   
>>>>
>>>> perhaps hook in via function query:
>>>>   dist(10.4,20.2,geoloc)
>>>>
>>>> And either manipulate the score with that and sort by score,
>>>>
>>>> q=+(foo bar)0 dist(10.4,20.2,geoloc)
>>>> sort=score asc
>>>>
>>>> or extend solr's sorting mechanisms to allow specifying a function to
>>>> 
> sort
>   
>>> 
>>>   
>> by.
>>   
>> 
>>>> sort="dist(10.4,20.2,geoloc) asc"
>>>>
>>>> -Yonik
>>>>
>>>> 
>>> 
>>>   
>> This email is confidential and may also be privileged. If you are not the
>> 
> intended recipient please notify us immediately by telephoning +44 (0)20
> 7452 5300 

RE: locallucene former custom-sort thread

[Sandeep Shetty]

yes i was thinking abt the same. 

i was searching for a radius of 25 miles. we get about 2500 results back for
the search. it seems like its storing all those geo results in cache and it
keeps on adding to it each time a geo request is made...

thanks for looking into it! 

Sandeep

-Original Message-
From: patrick o'leary [mailto:[EMAIL PROTECTED]
Sent: 28 September 2007 17:02
To: solr-user@lucene.apache.org
Subject: Re: locallucene former custom-sort thread


That's the latest. I was experimenting with caching, which might be the
problem.
I'll have a look, could you give me an idea of how large the radius was
and how many results were coming back.

Thanks
P

Sandeep Shetty wrote:
> Hi, i'm using local lucene, downloaded the latest zip file
> solr-example_s1.3_ls0.2.tgz
>
> is there a newer version available? 
>
> Thanks!
> Sandeep
>
> -Original Message-
> From: patrick o'leary [mailto:[EMAIL PROTECTED]
> Sent: 28 September 2007 16:08
> To: solr-user@lucene.apache.org
> Subject: locallucene former custom-sort thread
>
>
> Changing thread name;
>
> Are you using local lucene or local solr, and which version?
>
>
> P
>
> [EMAIL PROTECTED] wrote:
>   
>> i have been testing locallucene with our data for the last couple of
days.
>> one issue i faced with it is during when using geo sorting is that it
>> 
> seems
>   
>> to eat up all the memory, however big and become progressively slower,
>> finally after several requests (10 or so in my case) it throws up a
>> java.lang.OutOfMemoryError: Java heap space error.
>>
>> is there a way to get around this?
>>
>> -Original Message-
>> From: Jon Pierce [mailto:[EMAIL PROTECTED]
>> Sent: 28 September 2007 15:48
>> To: solr-user@lucene.apache.org
>> Subject: Re: custom sorting
>>
>>
>> Is the machinery in place to do this now (hook up a function query to
>> be used in sorting)?
>>
>> I'm trying to figure out what's the best way to do a distance sort:
>> custom comparator or function query.
>>
>> Using a custom comparator seems straightforward and reusable across
>> both the standard and dismax handlers.  But it also seems most likely
>> to impact performance (or at least require the most work/knowledge to
>> get right by minimizing calculations, caching, watching out for memory
>> leaks, etc.).  (Speaking of which, could anyone with more Lucene/Solr
>> experience than I comment on the performance characteristics of the
>> locallucene implementation mentioned on the list recently?  I've taken
>> a first look and it seems reasonable to me.)
>>
>> Using a function query, as Yonik suggests above, is another approach.
>> But to get a true sort, you have to boost the original query to zero?
>> How does this impact the results returned by the original query?  Will
>> the requirements (and boosts) of the original (now nested) query
>> remain intact, only sorted by the function?  Also, is there any way to
>> do this with the dismax handler?
>>
>> Thanks,
>> - Jon
>>
>> On 9/27/07, Yonik Seeley <[EMAIL PROTECTED]> wrote:
>>   
>> 
>>>> On 9/27/07, Erik Hatcher <[EMAIL PROTECTED]> wrote:
>>>> 
>>> 
>>>   
>>>>>> Using something like this, how would the custom SortComparatorSource
>>>>>> get a parameter from the request to use in sorting calculations?
>>>>>> 
>>>>   
>>>>
>>>> perhaps hook in via function query:
>>>>   dist(10.4,20.2,geoloc)
>>>>
>>>> And either manipulate the score with that and sort by score,
>>>>
>>>> q=+(foo bar)0 dist(10.4,20.2,geoloc)
>>>> sort=score asc
>>>>
>>>> or extend solr's sorting mechanisms to allow specifying a function to
>>>> 
> sort
>   
>>> 
>>>   
>> by.
>>   
>> 
>>>> sort="dist(10.4,20.2,geoloc) asc"
>>>>
>>>> -Yonik
>>>>
>>>> 
>>> 
>>>   
>> This email is confidential and may also be privileged. If you are not the
>> 
> intended recipient please notify us immediately by telephoning +44 (0)20
> 7452 5300 or email [EMAIL PROTECTED] You should not copy it or
use
> it for any purpose nor disclose its contents to any other person. Touch
> Local cannot accept liability for statements made which are clearly the
> sender's own and are not made on behalf of the firm.
>   
>> Touch Local Limited
>> Registered Number: 2885607
>> VAT Number: GB896112114
>> Cardinal Tower, 12 Farringdon Road, London EC1M 3NN
>> +44 (0)20 7452 5300
>>
>>   
>> 
>
>   

-- 

Patrick O'Leary

AOL Local Search Technologies
Phone: + 1 703 265 8763

You see, wire telegraph is a kind of a very, very long cat. You pull his
tail in New York and his head is meowing in Los Angeles.
 Do you understand this? 
And radio operates exactly the same way: you send signals here, they receive
them there. The only difference is that there is no cat.
  - Albert Einstein

View Patrick O Leary's LinkedIn profileView Patrick O Leary's profile
<http://www.linkedin.com/in/pjaol>

embeddedsolr and solrj index update request

[Sandeep Shetty]

Hi all,

i am using embeddedsolr and solrj to create and update a large index. however 
even after adding documents and commiting documents i.e.
server.add(solrDocs);
server.commit();
i am  not able to see the new documents in search results. I have to restart 
the server and run the same search to see the results. is there anything i am 
doing wrong here?

Thanks in advance for your help
Sandeep


This email is confidential and may also be privileged. If you are not the 
intended recipient please notify us immediately by telephoning +44 (0)20 7840 
4300 or email [EMAIL PROTECTED] You should not copy it or use it for any 
purpose nor disclose its contents to any other person. Touch Local cannot 
accept liability for statements made which are clearly the sender's own and are 
not made on behalf of the firm.

Touch Local Limited
Registered Number: 2885607
VAT Number: GB896112114
9th Floor, 89 Albert Embankment, London SE1 7TP
+44 (0)20 7840 4300

RE: embeddedsolr and solrj index update request

[Sandeep Shetty]

Hi Ryan

i am running solr 1.3

in my solrconfig.xml i can see  is that the right one?

Regards,
Sandeep

-Original Message-
From: Ryan Mckinley [mailto:[EMAIL PROTECTED]
Sent: 13 December 2007 17:43
To: solr-user@lucene.apache.org
Subject: Re: embeddedsolr and solrj index update request


What version solr are you running?
Do you have an updateRequestJandler registered to /update?

On Dec 13, 2007, at 8:36 AM, Sandeep Shetty <[EMAIL PROTECTED]
 > wrote:

> Hi all,
>
> i am using embeddedsolr and solrj to create and update a large
> index. however even after adding documents and commiting documents
> i.e.
> server.add(solrDocs);
> server.commit();
> i am  not able to see the new documents in search results. I have to
> restart the server and run the same search to see the results. is
> there anything i am doing wrong here?
>
> Thanks in advance for your help
> Sandeep
>
>
> This email is confidential and may also be privileged. If you are
> not the intended recipient please notify us immediately by
> telephoning +44 (0)20 7840 4300 or email [EMAIL PROTECTED]
> You should not copy it or use it for any purpose nor disclose its
> contents to any other person. Touch Local cannot accept liability
> for statements made which are clearly the sender's own and are not
> made on behalf of the firm.
>
> Touch Local Limited
> Registered Number: 2885607
> VAT Number: GB896112114
> 9th Floor, 89 Albert Embankment, London SE1 7TP
> +44 (0)20 7840 4300

This email is confidential and may also be privileged. If you are not the 
intended recipient please notify us immediately by telephoning +44 (0)20 7840 
4300 or email [EMAIL PROTECTED] You should not copy it or use it for any 
purpose nor disclose its contents to any other person. Touch Local cannot 
accept liability for statements made which are clearly the sender's own and are 
not made on behalf of the firm.

Touch Local Limited
Registered Number: 2885607
VAT Number: GB896112114
9th Floor, 89 Albert Embankment, London SE1 7TP
+44 (0)20 7840 4300

matching exact/whole phrase

[Sandeep Shetty]

Hi people,

I am looking to provide exact phrase match, along with the full text search 
with solr.  I want to achieve the same effect in solr rather than use a 
separate SQL query. I want to do the following as an example

The indexed field has the text "car repair" (without the double quotes)  for a 
document and I want this document to come in the search result only if someone 
searches for "car repair". The document should not show up for "repair" and 
"car" searches.

Is it possible to do this type of exact phrase matching if needed with solr 
itself?

Thanks in advance

Regards,
Sandeep




Sandeep Shetty
Technical Development Manager

Touch Local
89 Albert Embankment, London, SE1 7TP, UK
D: 020 7840 4335
E: [EMAIL PROTECTED]
T: 020 7840 4300
F: 020 7840 4301 

This email is confidential and may also be privileged. If you are not the 
intended recipient please notify us immediately by calling 020 7840 4300 or 
email [EMAIL PROTECTED] You should not copy it or use it for any purpose nor 
disclose its contents to any other person. Touch Local Ltd cannot accept 
liability for statements made which are clearly the sender's own and are not 
made on behalf of the firm.
Registered in England and Wales. Registration Number: 2885607 VAT Number: 
GB896112114

Help to save some trees. Print e-mails only if you really need to.

RE: matching exact/whole phrase

[Sandeep Shetty]

That was the answer I was looking for, I will try that one out

Thanks Daniel

-Original Message-
From: Daniel Papasian [mailto:[EMAIL PROTECTED]
Sent: 01 April 2008 16:03
To: solr-user@lucene.apache.org
Subject: Re: matching exact/whole phrase

Sandeep Shetty wrote:
> Hi people,
>
> I am looking to provide exact phrase match, along with the full text
> search with solr.  I want to achieve the same effect in solr rather
> than use a separate SQL query. I want to do the following as an
> example
>
> The indexed field has the text "car repair" (without the double
> quotes)  for a document and I want this document to come in the
> search result only if someone searches for "car repair". The document
> should not show up for "repair" and "car" searches.
>
> Is it possible to do this type of exact phrase matching if needed
> with solr itself?

It sounds like you want to do an exact string match, and not a text
match, so I don't think there's anything complex you'd need to do...
just store the field with "car repair" as type="string" and do all of
the literal searches you want.

But if you are working off a field that contains something beyond the
exact match of what you want to search for, you'll just need to define a
new field type and use only the analysis filters that you need, and
you'll have to think more about what you need if that's the case.

Daniel

Sandeep Shetty
Technical Development Manager

Touch Local
89 Albert Embankment, London, SE1 7TP, UK
D: 020 7840 4335
E: [EMAIL PROTECTED]
T: 020 7840 4300
F: 020 7840 4301 

This email is confidential and may also be privileged. If you are not the 
intended recipient please notify us immediately by calling 020 7840 4300 or 
email [EMAIL PROTECTED] You should not copy it or use it for any purpose nor 
disclose its contents to any other person. Touch Local Ltd cannot accept 
liability for statements made which are clearly the sender's own and are not 
made on behalf of the firm.
Registered in England and Wales. Registration Number: 2885607 VAT Number: 
GB896112114

Help to save some trees. Print e-mails only if you really need to.