Custom Sorting
Hi All,
Currently we are having an application which uses Lucene for text search and
we are in the process of migrating to Solr.
In our Lucene code we have the following way of using the sort criteria
Sort sort = new Sort(new SortField[]
{ SortField.FIELD_SCORE, new SortField(customValue, SortField.FLOAT,
true) });
indexSearcher.search(q, sort)
and another code snippet is
Sort sort = new Sort(new SortField(customValue, customComparator))
indexSearcher.search(q, sort)
In these two scenarios, how should I configure the schema to have my own
sort definition for the query.
I couldnt find any documentation which describes this query time sort
definition in any of the documentation. Can any of you please throw some
light on this.
Thanks and Regards
Palasseri
Re: How to read values of a field efficiently
On Mon, 2007-07-30 at 00:30 -0700, Chris Hostetter wrote: > : Is it possible to get the values from the ValueSource (or from > : getFieldCacheCounts) sorted by its natural order (from lowest to > : highest values)? > > well, an inverted term index is already a data structure listing terms > from lowest to highest and the associated documents -- so if you want to > iterate from low to high between a range and find matching docs you should > just use hte TermEnum Ok. Unfortunately I don't see how I can get a TermEnum for a specific field (e.g. "price")... I tried TermEnum te = searcher.getReader().terms(new Term(field, "")); but this returns also terms for several other fields. Is it possible at all to get a TermEnum for a specific field? Then if I had this TermEnum, how can I check if a Term is in my DocSet? In other words, I would like to read Terms for a specific field from my DocSet - so that I could determine all price terms for my DocSet. Is there a way to achieve this? Thanx in advance, cheers, Martin > -- the whole point of the FieldCache (and > FieldCacheSource) is to have a "reverse inverted index" so you can quickly > fetch the indexed value if you know the docId. > > perhaps you should elaborate a little more on what it is you are trying to > do so we can help you figure out how to do it more efficinelty ... i know > you mentioend computing price ranges in your first message ... but you > also didn't post any clear code about that part of your problem, just that > the *other* part of your code that iterated over every doc was too slow > ... perhaps you shouldn't be iterating over every doc to figure out your > ranges .. perhaps you can iterate over the terms themselves? > > > hang on ... rereading your first message i just noticed something i > definitely didn't spot before... > > >> Fairly long: getFieldCacheCounts for the cat field takes ~70 ms > >> for the second request, while reading prices takes ~600 ms. > > ...i clearly missed this, and fixated on your assertion that your reading > of field values took longer then the stock methods -- but you're not just > comparing the time needed byu different methods, you're also timing > different fields. > > this actually makes a lot of sense since there are probably a lot fewer > unique values for the cat field, so there are a lot fewer discrete values > to deal with when computing counts. > > > > > -Hoss > -- Martin Grotzke http://www.javakaffee.de/blog/ signature.asc Description: This is a digitally signed message part
Start up script for solr?
Hi, Sorry that this is not strictly a solr specific question - I wonder if anyone has a script to start solr on Linux when the system boots up? Or better yet, supports shutdown and restart? -- Thanks, Jack
Re: Start up script for solr?
I blogged about it last month, here ya go. http://www.digital39.com/programming/solr-chkconfig-and-startstop-scripts/2007/07/304/ - Pete On 8/19/07, Jack L <[EMAIL PROTECTED]> wrote: > Hi, > > Sorry that this is not strictly a solr specific question - > > I wonder if anyone has a script to start solr on Linux when the > system boots up? Or better yet, supports shutdown and restart? > > -- > Thanks, > Jack > >
Re: Start up script for solr?
I forgot to mention, that is for a RHEL box, but can easily be adapted. It will work like the standard scripts for RHEL /etc/init.d/solr start /etc/init.d/solr stop /etc/init.d/solr restart or you can just run the solr.start and solr.stop scripts individually On 8/19/07, Peter Manis <[EMAIL PROTECTED]> wrote: > I blogged about it last month, here ya go. > > http://www.digital39.com/programming/solr-chkconfig-and-startstop-scripts/2007/07/304/ > > - Pete > > On 8/19/07, Jack L <[EMAIL PROTECTED]> wrote: > > Hi, > > > > Sorry that this is not strictly a solr specific question - > > > > I wonder if anyone has a script to start solr on Linux when the > > system boots up? Or better yet, supports shutdown and restart? > > > > -- > > Thanks, > > Jack > > > > >
Re: Re[2]: Start up script for solr?
Interesting, it worked fine on the server. Try moving the -stop at the end of the line to just before the -jar. - Pete On 8/19/07, Jack L <[EMAIL PROTECTED]> wrote: > Hello Peter, > > Many thanks! > > solr.start works fine but I'm getting an error with solr.stop and solr is not > being stopped: > (I've replaced my app dir with /opt/directory in the log: > > $ /etc/init.d/solr stop > Stopping Solr... java.net.BindException: Address already in use > WARN: Not listening on monitor port: 8079 > 2007-08-19 15:23:02.533::INFO: Logging to STDERR via > org.mortbay.log.StdErrLog > 2007-08-19 15:23:02.563::WARN: EXCEPTION > java.io.FileNotFoundException: /opt/directory/-stop (No such file or > directory) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.(FileInputStream.java:106) > at java.io.FileInputStream.(FileInputStream.java:66) > at > sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:70) > at > sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:161) > at > com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:653) > at > com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771) > at > com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737) > at > com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107) > at > com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205) > at > com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522) > at javax.xml.parsers.SAXParser.parse(SAXParser.java:395) > at org.mortbay.xml.XmlParser.parse(XmlParser.java:188) > at org.mortbay.xml.XmlParser.parse(XmlParser.java:204) > at org.mortbay.xml.XmlConfiguration.(XmlConfiguration.java:100) > at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:916) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.mortbay.start.Main.invokeMain(Main.java:183) > at org.mortbay.start.Main.start(Main.java:497) > at org.mortbay.start.Main.main(Main.java:115) > OK > > > -- > Best regards, > Jack > > Sunday, August 19, 2007, 11:43:16 AM, you wrote: > > > I blogged about it last month, here ya go. > > > http://www.digital39.com/programming/solr-chkconfig-and-startstop-scripts/2007/07/304/ > > > - Pete > > > On 8/19/07, Jack L <[EMAIL PROTECTED]> wrote: > >> Hi, > >> > >> Sorry that this is not strictly a solr specific question - > >> > >> I wonder if anyone has a script to start solr on Linux when the > >> system boots up? Or better yet, supports shutdown and restart? > >> > >> -- > >> Thanks, > >> Jack > >> > >> > >
RE: How to read values of a field efficiently
Hello,
>
> On Mon, 2007-07-30 at 00:30 -0700, Chris Hostetter wrote:
> > : Is it possible to get the values from the ValueSource (or from
> > : getFieldCacheCounts) sorted by its natural order (from lowest to
> > : highest values)?
> >
> > well, an inverted term index is already a data structure
> listing terms
> > from lowest to highest and the associated documents -- so
> if you want to
> > iterate from low to high between a range and find matching
> docs you should
> > just use hte TermEnum
> Ok. Unfortunately I don't see how I can get a TermEnum for a specific
> field (e.g. "price")... I tried
>
> TermEnum te = searcher.getReader().terms(new Term(field, ""));
>
> but this returns also terms for several other fields.
correct, see
http://lucene.zones.apache.org:8080/hudson/job/Lucene-Nightly/javadoc/org/apache/lucene/index/IndexReader.html#terms()
> Is it possible at all to get a TermEnum for a specific field?
AFAIK not directly. Normally, I use something like:
TermEnum terms = searcher.getReader().terms(new Term(field, ""));
while (terms.term() != null && terms.term().field() == field){
//do things
terms.next();
}
>
> Then if I had this TermEnum, how can I check if a Term is in my
> DocSet? In other words, I would like to read Terms for a specific
> field from my DocSet - so that I could determine all price terms
> for my DocSet.
Is your DocSet some sort of filter? if so, in your while loop you can fill a
new Filter, like
BitSet docFilter = new BitSet(reader.maxDoc());
and in the while loop:
docs.seek(terms);
while (docs.next()) {
docFilter.set(docs.doc());
}
If your DocSet is not a BitSet you might be able to construct one for it,
Regards Ard
>
> Is there a way to achieve this?
>
> Thanx in advance,
> cheers,
> Martin
>
>
> > -- the whole point of the FieldCache (and
> > FieldCacheSource) is to have a "reverse inverted index" so
> you can quickly
> > fetch the indexed value if you know the docId.
> >
> > perhaps you should elaborate a little more on what it is
> you are trying to
> > do so we can help you figure out how to do it more
> efficinelty ... i know
> > you mentioend computing price ranges in your first message
> ... but you
> > also didn't post any clear code about that part of your
> problem, just that
> > the *other* part of your code that iterated over every doc
> was too slow
> > ... perhaps you shouldn't be iterating over every doc to
> figure out your
> > ranges .. perhaps you can iterate over the terms themselves?
> >
> >
> > hang on ... rereading your first message i just noticed something i
> > definitely didn't spot before...
> >
> > >> Fairly long: getFieldCacheCounts for the cat field takes ~70 ms
> > >> for the second request, while reading prices takes ~600 ms.
> >
> > ...i clearly missed this, and fixated on your assertion
> that your reading
> > of field values took longer then the stock methods -- but
> you're not just
> > comparing the time needed byu different methods, you're also timing
> > different fields.
> >
> > this actually makes a lot of sense since there are probably
> a lot fewer
> > unique values for the cat field, so there are a lot fewer
> discrete values
> > to deal with when computing counts.
> >
> >
> >
> >
> > -Hoss
> >
> --
> Martin Grotzke
> http://www.javakaffee.de/blog/
>
Re[4]: Start up script for solr?
If I remove -stop, it'll start another instance or solr and it will fail because the port has been taken by the first solr instance ... -- Best regards, Jack Sunday, August 19, 2007, 12:35:00 PM, you wrote: > Interesting, it worked fine on the server. Try moving the -stop at > the end of the line to just before the -jar. > - Pete > On 8/19/07, Jack L <[EMAIL PROTECTED]> wrote: >> Hello Peter, >> >> Many thanks! >> >> solr.start works fine but I'm getting an error with solr.stop and solr is >> not being stopped: >> (I've replaced my app dir with /opt/directory in the log: >> >> $ /etc/init.d/solr stop >> Stopping Solr... java.net.BindException: Address already in use >> WARN: Not listening on monitor port: 8079 >> 2007-08-19 15:23:02.533::INFO: Logging to STDERR via >> org.mortbay.log.StdErrLog >> 2007-08-19 15:23:02.563::WARN: EXCEPTION >> java.io.FileNotFoundException: /opt/directory/-stop (No such file or >> directory) >> at java.io.FileInputStream.open(Native Method) >> at java.io.FileInputStream.(FileInputStream.java:106) >> at java.io.FileInputStream.(FileInputStream.java:66) >> at >> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:70) >> at >> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:161) >> at >> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:653) >> at >> com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186) >> at >> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771) >> at >> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737) >> at >> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107) >> at >> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205) >> at >> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522) >> at javax.xml.parsers.SAXParser.parse(SAXParser.java:395) >> at org.mortbay.xml.XmlParser.parse(XmlParser.java:188) >> at org.mortbay.xml.XmlParser.parse(XmlParser.java:204) >> at >> org.mortbay.xml.XmlConfiguration.(XmlConfiguration.java:100) >> at >> org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:916) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.mortbay.start.Main.invokeMain(Main.java:183) >> at org.mortbay.start.Main.start(Main.java:497) >> at org.mortbay.start.Main.main(Main.java:115) >> OK >> >> >> -- >> Best regards, >> Jack >> >> Sunday, August 19, 2007, 11:43:16 AM, you wrote: >> >> > I blogged about it last month, here ya go. >> >> > >> http://www.digital39.com/programming/solr-chkconfig-and-startstop-scripts/2007/07/304/ >> >> > - Pete >> >> > On 8/19/07, Jack L <[EMAIL PROTECTED]> wrote: >> >> Hi, >> >> >> >> Sorry that this is not strictly a solr specific question - >> >> >> >> I wonder if anyone has a script to start solr on Linux when the >> >> system boots up? Or better yet, supports shutdown and restart? >> >> >> >> -- >> >> Thanks, >> >> Jack >> >> >> >> >> >>
Re[4]: Start up script for solr?
Sorry. If I move it to before -jar, I get: Unrecognized option: -stop Could not create the Java virtual machine. Sunday, August 19, 2007, 12:35:00 PM, you wrote: > Interesting, it worked fine on the server. Try moving the -stop at > the end of the line to just before the -jar. > - Pete > On 8/19/07, Jack L <[EMAIL PROTECTED]> wrote: >> Hello Peter, >> >> Many thanks! >> >> solr.start works fine but I'm getting an error with solr.stop and solr is >> not being stopped: >> (I've replaced my app dir with /opt/directory in the log: >> >> $ /etc/init.d/solr stop >> Stopping Solr... java.net.BindException: Address already in use >> WARN: Not listening on monitor port: 8079 >> 2007-08-19 15:23:02.533::INFO: Logging to STDERR via >> org.mortbay.log.StdErrLog >> 2007-08-19 15:23:02.563::WARN: EXCEPTION >> java.io.FileNotFoundException: /opt/directory/-stop (No such file or >> directory) >> at java.io.FileInputStream.open(Native Method) >> at java.io.FileInputStream.(FileInputStream.java:106) >> at java.io.FileInputStream.(FileInputStream.java:66) >> at >> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:70) >> at >> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:161) >> at >> com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:653) >> at >> com.sun.org.apache.xerces.internal.impl.XMLVersionDetector.determineDocVersion(XMLVersionDetector.java:186) >> at >> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771) >> at >> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737) >> at >> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:107) >> at >> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1205) >> at >> com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:522) >> at javax.xml.parsers.SAXParser.parse(SAXParser.java:395) >> at org.mortbay.xml.XmlParser.parse(XmlParser.java:188) >> at org.mortbay.xml.XmlParser.parse(XmlParser.java:204) >> at >> org.mortbay.xml.XmlConfiguration.(XmlConfiguration.java:100) >> at >> org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:916) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> at java.lang.reflect.Method.invoke(Method.java:597) >> at org.mortbay.start.Main.invokeMain(Main.java:183) >> at org.mortbay.start.Main.start(Main.java:497) >> at org.mortbay.start.Main.main(Main.java:115) >> OK >> >> >> -- >> Best regards, >> Jack >> >> Sunday, August 19, 2007, 11:43:16 AM, you wrote: >> >> > I blogged about it last month, here ya go. >> >> > >> http://www.digital39.com/programming/solr-chkconfig-and-startstop-scripts/2007/07/304/ >> >> > - Pete >> >> > On 8/19/07, Jack L <[EMAIL PROTECTED]> wrote: >> >> Hi, >> >> >> >> Sorry that this is not strictly a solr specific question - >> >> >> >> I wonder if anyone has a script to start solr on Linux when the >> >> system boots up? Or better yet, supports shutdown and restart? >> >> >> >> -- >> >> Thanks, >> >> Jack >> >> >> >> >> >>
Re[4]: Start up script for solr?
Actually it's --stop. Thanks! > Interesting, it worked fine on the server. Try moving the -stop at > the end of the line to just before the -jar. > - Pete
Re: Re[4]: Start up script for solr?
Sorry about that, I left out the 2nd dash when I added it to the blog. Glad it is working now On 8/19/07, Jack L <[EMAIL PROTECTED]> wrote: > Actually it's --stop. Thanks! > > > Interesting, it worked fine on the server. Try moving the -stop at > > the end of the line to just before the -jar. > > > - Pete > > >
Re: how to retrieve all the documents in an index?
BTW, Hoss, is there a default order for the documents returned by running this query? thanks, -Hui On 8/16/07, Chris Hostetter <[EMAIL PROTECTED]> wrote: > > > : Any of you know whether the new "q:*.*" query performs better than the > : get-around solutions like using a ranged query? I would guess so, but I > : haven't looked into the Lucene implementation. > > it's faster -- it has almost no work to do relative the range query > version. > > > > -Hoss > > -- Regards, -Hui
