Re: Nwebie Question on boosting
Thanks Rob. Can you plz provide some sample documents (lucene) for title bassed boosting? -- View this message in context: http://www.nabble.com/Nwebie-Question-on-boosting-tp20950286p20952532.html Sent from the Solr - User mailing list archive at Nabble.com.
Taxonomy Support on Solr
Hi, Any plans of supporting user-defined classifications on Solr? Is there any component which returns all the children of a node (till the leaf node) when I search for any node? May be this would help: Say I have a few SolrDocuments classified as: A B--C 123 8--9 (I.e A has 2 child nodes B and C. B has 3 child nodes 1,2,3 and C has 2 child nodes 8,9) When my search criteria matches B, my results should contain B as well as 1,2 and 3 too. Search for A would return all the nodes mentioned above. -Kumar
Make it more performant - solr 1.3 - 1200msec respond time.
Hi, I'm doing a stress test on solr. I've around 8,5M of doc, the size of my data's directory is 5,6G. I've indexed again my data to make it faster, and applied all the last patch. My index data store just two field : id and text (which is a copy of three fiels) But I still think it's very long, what do you think? For 50request/sec during 40mn, my average respond time : 1235msec. 49430request. When I make this test with 100request second during 10mn and 10 other minutes with 50 request : my average respond time is 1600msec. Don't you think it's a bit long. Should I partition this index more ? or what should I do to make this work faster. I can read post with people who have just 300msec request for 300Go of index partionned ? My request for collecting all this book is quite complex and have lot of table linked together, maybe it would be faster if I create a csv file ? The server that I'm using for the test has 8G of memory. 4CPU : Intel(R) Xeon(R) CPU5160 @ 3.00GHz Tomcat55 : -Xms2000m -Xmx4000m Solr 1.3. What can I modify to make it more performant ? memory, indexation ...? Does it can come from my request to the mysql database which is too much linked ? Thanks a lot for your help, Johanna -- View this message in context: http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20953079.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Nwebie Question on boosting
Hi, ayyanar wrote: > > Thanks Rob. Can you plz provide some sample documents (lucene) for title > bassed boosting? > I'm not sure about the Lucene part (this is the Solr mailing list after all), but if you want index time boosting of certain fields, you have to add documents like this: 123 My boosted field Here goes the text, yadda yadda In this document the title-field will be boosted with a factor of 2, if it has norms enabled. (omitNorms = false) You can read more about this here: http://wiki.apache.org/solr/UpdateXmlMessages#head-63c170cad521de8d1e9be0d76025774cf3b66dfc -- View this message in context: http://www.nabble.com/Nwebie-Question-on-boosting-tp20950286p20953736.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Make it more performant - solr 1.3 - 1200msec respond time.
Are each of those queries unique? First time queries are slower. They are cached by Solr and the same query again will return results very quickly because it won't need to hit the file system. On Thu, Dec 11, 2008 at 4:08 PM, sunnyfr <[EMAIL PROTECTED]> wrote: > > Hi, > > I'm doing a stress test on solr. > I've around 8,5M of doc, the size of my data's directory is 5,6G. > > I've indexed again my data to make it faster, and applied all the last > patch. > My index data store just two field : id and text (which is a copy of three > fiels) > But I still think it's very long, what do you think? > > For 50request/sec during 40mn, my average respond time : 1235msec. > 49430request. > > When I make this test with 100request second during 10mn and 10 other > minutes with 50 request : my average respond time is 1600msec. Don't you > think it's a bit long. > > Should I partition this index more ? or what should I do to make this work > faster. > I can read post with people who have just 300msec request for 300Go of > index > partionned ? > My request for collecting all this book is quite complex and have lot of > table linked together, maybe it would be faster if I create a csv file ? > > The server that I'm using for the test has 8G of memory. > 4CPU : Intel(R) Xeon(R) CPU5160 @ 3.00GHz > Tomcat55 : -Xms2000m -Xmx4000m > Solr 1.3. > > What can I modify to make it more performant ? memory, indexation ...? > Does it can come from my request to the mysql database which is too much > linked ? > > Thanks a lot for your help, > Johanna > > > -- > View this message in context: > http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20953079.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- Regards, Shalin Shekhar Mangar.
Re: Make it more performant - solr 1.3 - 1200msec respond time.
So according to you and everything explained in my post, I did my best to optimize it ? Yes it's unique queries. I will try it again and activate cache. What you mean by hit the file system? thanks a lot Shalin Shekhar Mangar wrote: > > Are each of those queries unique? > > First time queries are slower. They are cached by Solr and the same query > again will return results very quickly because it won't need to hit the > file > system. > > On Thu, Dec 11, 2008 at 4:08 PM, sunnyfr <[EMAIL PROTECTED]> wrote: > >> >> Hi, >> >> I'm doing a stress test on solr. >> I've around 8,5M of doc, the size of my data's directory is 5,6G. >> >> I've indexed again my data to make it faster, and applied all the last >> patch. >> My index data store just two field : id and text (which is a copy of >> three >> fiels) >> But I still think it's very long, what do you think? >> >> For 50request/sec during 40mn, my average respond time : 1235msec. >> 49430request. >> >> When I make this test with 100request second during 10mn and 10 other >> minutes with 50 request : my average respond time is 1600msec. Don't you >> think it's a bit long. >> >> Should I partition this index more ? or what should I do to make this >> work >> faster. >> I can read post with people who have just 300msec request for 300Go of >> index >> partionned ? >> My request for collecting all this book is quite complex and have lot of >> table linked together, maybe it would be faster if I create a csv file ? >> >> The server that I'm using for the test has 8G of memory. >> 4CPU : Intel(R) Xeon(R) CPU5160 @ 3.00GHz >> Tomcat55 : -Xms2000m -Xmx4000m >> Solr 1.3. >> >> What can I modify to make it more performant ? memory, indexation ...? >> Does it can come from my request to the mysql database which is too much >> linked ? >> >> Thanks a lot for your help, >> Johanna >> >> >> -- >> View this message in context: >> http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20953079.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20954392.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Sum of Fields and Record Count
Hi Otis, Thanks for the info and help. I started reading up about it (on Markmail, nice site), and it looks like there is some activity to put it into 1.4. I will try and apply the patch, and see how that works. It seems like a couple of people are using it in a production environment already, with out grief. So that is a good thing. -John On Dec 11, 2008, at 1:24 AM, Otis Gospodnetic wrote: Hi John, It's not in the current release, but the chances are it will make it into 1.4. You can try one of the recent patches and apply it to your Solr 1.3 sources. Check list archives for more discussion, this field collapsing was just discussed again today/yesterday. markmail.org is a good one. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: John Martyniak <[EMAIL PROTECTED]> To: solr-user@lucene.apache.org Sent: Wednesday, December 10, 2008 10:51:57 PM Subject: Re: Sum of Fields and Record Count Otis, Thanks for the information. It looks like the field collapsing is similar to what I am looking. But is that in the current release? Is it stable? Is there anyway to do it in Solr 1.3? -John On Dec 10, 2008, at 9:59 PM, Otis Gospodnetic wrote: Hi John, This sounds a lot like field collapsing functionality that a few people are working on in SOLR-236: https://issues.apache.org/jira/browse/SOLR-236 Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message From: John Martyniak To: solr-user@lucene.apache.org Sent: Wednesday, December 10, 2008 6:16:21 PM Subject: Sum of Fields and Record Count Hi, I am a new solr user. I have an application that I would like to show the results but one result may be the part of larger set of results. So for example result #1 might also have 10 other results that are part of the same data set. Hopefully this makes sense. What I would like to find out is if there is a way within Solr to show the result that matched with the query, and then to also show that this result is part of a collection of 10 items. I have thought about doing it using some sort of external process that runs, and with doing multiple queries, so get the list of items and then query against each item. But those don't seem elegant. So I would like to find out if there is a way to do it within Solr that is a little more elegant, and hopefully without having to write additional code. Thank you in advance for the help. -John
Re: Make it more performant - solr 1.3 - 1200msec respond time.
Actually I just notices, lot of request didn"t bring back correct answer, but " No read Solr server available" so my jmeter didn't take that for an error. Obviously out of memory, and a file gc.log is created with : 0.054: [GC [PSYoungGen: 5121K->256K(298688K)] 5121K->256K(981376K), 0.0020630 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 0.056: [Full GC (System) [PSYoungGen: 256K->0K(298688K)] [PSOldGen: 0K->180K(682688K)] 256K->180K(981376K) [PSPermGen: 3002K->3002K(21248K)], 0.0055170 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] so far my tomcat55 file is configurate like that : JAVA_OPTS="-Xms1000m -Xmx4000m -XX:+HeapDumpOnOutOfMemoryError -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps" Thanks for your help Shalin Shekhar Mangar wrote: > > Are each of those queries unique? > > First time queries are slower. They are cached by Solr and the same query > again will return results very quickly because it won't need to hit the > file > system. > > On Thu, Dec 11, 2008 at 4:08 PM, sunnyfr <[EMAIL PROTECTED]> wrote: > >> >> Hi, >> >> I'm doing a stress test on solr. >> I've around 8,5M of doc, the size of my data's directory is 5,6G. >> >> I've indexed again my data to make it faster, and applied all the last >> patch. >> My index data store just two field : id and text (which is a copy of >> three >> fiels) >> But I still think it's very long, what do you think? >> >> For 50request/sec during 40mn, my average respond time : 1235msec. >> 49430request. >> >> When I make this test with 100request second during 10mn and 10 other >> minutes with 50 request : my average respond time is 1600msec. Don't you >> think it's a bit long. >> >> Should I partition this index more ? or what should I do to make this >> work >> faster. >> I can read post with people who have just 300msec request for 300Go of >> index >> partionned ? >> My request for collecting all this book is quite complex and have lot of >> table linked together, maybe it would be faster if I create a csv file ? >> >> The server that I'm using for the test has 8G of memory. >> 4CPU : Intel(R) Xeon(R) CPU5160 @ 3.00GHz >> Tomcat55 : -Xms2000m -Xmx4000m >> Solr 1.3. >> >> What can I modify to make it more performant ? memory, indexation ...? >> Does it can come from my request to the mysql database which is too much >> linked ? >> >> Thanks a lot for your help, >> Johanna >> >> >> -- >> View this message in context: >> http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20953079.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> >> > > > -- > Regards, > Shalin Shekhar Mangar. > > -- View this message in context: http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20955210.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Make it more performant - solr 1.3 - 1200msec respond time.
Ok sorry I just add the parameter -XX:+UseParallelGC and it seems to don't go
oom.
sunnyfr wrote:
>
> Actually I just notices, lot of request didn"t bring back correct answer,
> but " No read Solr server available" so my jmeter didn't take that for an
> error. Obviously out of memory, and a file gc.log is created with :
> 0.054: [GC [PSYoungGen: 5121K->256K(298688K)] 5121K->256K(981376K),
> 0.0020630 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 0.056: [Full GC (System) [PSYoungGen: 256K->0K(298688K)] [PSOldGen:
> 0K->180K(682688K)] 256K->180K(981376K) [PSPermGen: 3002K->3002K(21248K)],
> 0.0055170 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>
> so far my tomcat55 file is configurate like that :
> JAVA_OPTS="-Xms1000m -Xmx4000m -XX:+HeapDumpOnOutOfMemoryError
> -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
>
> My error:
> Dec 11 14:16:27 solr-test jsvc.exec[30653]: Dec 11, 2008 2:16:27 PM
> org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
> path=/admin/ping params={} hits=0 status=0 QTime=1 Dec 11, 2008 2:16:27
> PM org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
> path=/admin/ping params={} status=0 QTime=1 Dec 11, 2008 2:16:27 PM
> org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
> path=/admin/ping params={} hits=0 status=0 QTime=1 Dec 11, 2008 2:16:27
> PM org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
> path=/admin/ping params={} status=0 QTime=2
> Dec 11 14:16:27 solr-test jsvc.exec[30653]: java.lang.OutOfMemoryError: GC
> overhead limit exceeded Dumping heap to java_pid30655.hprof ...
>
>
>
>
> Thanks for your help
>
>
>
> Shalin Shekhar Mangar wrote:
>>
>> Are each of those queries unique?
>>
>> First time queries are slower. They are cached by Solr and the same query
>> again will return results very quickly because it won't need to hit the
>> file
>> system.
>>
>> On Thu, Dec 11, 2008 at 4:08 PM, sunnyfr <[EMAIL PROTECTED]> wrote:
>>
>>>
>>> Hi,
>>>
>>> I'm doing a stress test on solr.
>>> I've around 8,5M of doc, the size of my data's directory is 5,6G.
>>>
>>> I've indexed again my data to make it faster, and applied all the last
>>> patch.
>>> My index data store just two field : id and text (which is a copy of
>>> three
>>> fiels)
>>> But I still think it's very long, what do you think?
>>>
>>> For 50request/sec during 40mn, my average respond time : 1235msec.
>>> 49430request.
>>>
>>> When I make this test with 100request second during 10mn and 10 other
>>> minutes with 50 request : my average respond time is 1600msec. Don't you
>>> think it's a bit long.
>>>
>>> Should I partition this index more ? or what should I do to make this
>>> work
>>> faster.
>>> I can read post with people who have just 300msec request for 300Go of
>>> index
>>> partionned ?
>>> My request for collecting all this book is quite complex and have lot of
>>> table linked together, maybe it would be faster if I create a csv file ?
>>>
>>> The server that I'm using for the test has 8G of memory.
>>> 4CPU : Intel(R) Xeon(R) CPU5160 @ 3.00GHz
>>> Tomcat55 : -Xms2000m -Xmx4000m
>>> Solr 1.3.
>>>
>>> What can I modify to make it more performant ? memory, indexation ...?
>>> Does it can come from my request to the mysql database which is too much
>>> linked ?
>>>
>>> Thanks a lot for your help,
>>> Johanna
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20953079.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
>
--
View this message in context:
http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20955479.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Make it more performant - solr 1.3 - 1200msec respond time.
Actually I still have this error : " No read Solr server available "
sunnyfr wrote:
>
> Ok sorry I just add the parameter -XX:+UseParallelGC and it seems to don't
> go oom.
>
>
>
>
> sunnyfr wrote:
>>
>> Actually I just notices, lot of request didn"t bring back correct answer,
>> but " No read Solr server available" so my jmeter didn't take that for an
>> error. Obviously out of memory, and a file gc.log is created with :
>> 0.054: [GC [PSYoungGen: 5121K->256K(298688K)] 5121K->256K(981376K),
>> 0.0020630 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>> 0.056: [Full GC (System) [PSYoungGen: 256K->0K(298688K)] [PSOldGen:
>> 0K->180K(682688K)] 256K->180K(981376K) [PSPermGen: 3002K->3002K(21248K)],
>> 0.0055170 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>>
>> so far my tomcat55 file is configurate like that :
>> JAVA_OPTS="-Xms1000m -Xmx4000m -XX:+HeapDumpOnOutOfMemoryError
>> -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
>>
>> My error:
>> Dec 11 14:16:27 solr-test jsvc.exec[30653]: Dec 11, 2008 2:16:27 PM
>> org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
>> path=/admin/ping params={} hits=0 status=0 QTime=1 Dec 11, 2008 2:16:27
>> PM org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
>> path=/admin/ping params={} status=0 QTime=1 Dec 11, 2008 2:16:27 PM
>> org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
>> path=/admin/ping params={} hits=0 status=0 QTime=1 Dec 11, 2008 2:16:27
>> PM org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
>> path=/admin/ping params={} status=0 QTime=2
>> Dec 11 14:16:27 solr-test jsvc.exec[30653]: java.lang.OutOfMemoryError:
>> GC overhead limit exceeded Dumping heap to java_pid30655.hprof ...
>>
>>
>>
>>
>> Thanks for your help
>>
>>
>>
>> Shalin Shekhar Mangar wrote:
>>>
>>> Are each of those queries unique?
>>>
>>> First time queries are slower. They are cached by Solr and the same
>>> query
>>> again will return results very quickly because it won't need to hit the
>>> file
>>> system.
>>>
>>> On Thu, Dec 11, 2008 at 4:08 PM, sunnyfr <[EMAIL PROTECTED]> wrote:
>>>
Hi,
I'm doing a stress test on solr.
I've around 8,5M of doc, the size of my data's directory is 5,6G.
I've indexed again my data to make it faster, and applied all the last
patch.
My index data store just two field : id and text (which is a copy of
three
fiels)
But I still think it's very long, what do you think?
For 50request/sec during 40mn, my average respond time : 1235msec.
49430request.
When I make this test with 100request second during 10mn and 10 other
minutes with 50 request : my average respond time is 1600msec. Don't
you
think it's a bit long.
Should I partition this index more ? or what should I do to make this
work
faster.
I can read post with people who have just 300msec request for 300Go of
index
partionned ?
My request for collecting all this book is quite complex and have lot
of
table linked together, maybe it would be faster if I create a csv file
?
The server that I'm using for the test has 8G of memory.
4CPU : Intel(R) Xeon(R) CPU5160 @ 3.00GHz
Tomcat55 : -Xms2000m -Xmx4000m
Solr 1.3.
What can I modify to make it more performant ? memory, indexation ...?
Does it can come from my request to the mysql database which is too
much
linked ?
Thanks a lot for your help,
Johanna
--
View this message in context:
http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20953079.html
Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>> --
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>>
>>
>>
>
>
--
View this message in context:
http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20955609.html
Sent from the Solr - User mailing list archive at Nabble.com.
Building Solr from Source
Hi, I have downloaded Maven 2.0.9, and tried to build using "mvn clean install" and "mvn install", nothing works. Can somebody tell me how to build solr from source? I am trying to build the 1.3 source. thank you very much, -John
Re: Building Solr from Source
Solr uses ant for build install ant On Thu, Dec 11, 2008 at 7:13 PM, John Martyniak <[EMAIL PROTECTED]> wrote: > Hi, > > I have downloaded Maven 2.0.9, and tried to build using "mvn clean install" > and "mvn install", nothing works. > > Can somebody tell me how to build solr from source? I am trying to build > the 1.3 source. > > thank you very much, > > -John > -- --Noble Paul
Re: Dynamic Boosting at query time with boost value as another fieldvalue
Take a look at FunctionQuery support in Solr: http://wiki.apache.org/solr/FunctionQuery http://wiki.apache.org/solr/SolrRelevancyFAQ#head-b1b1cdedcb9cd9bfd9c994709b4d7e540359b1fd On Thu, Dec 11, 2008 at 7:01 PM, Pooja Verlani <[EMAIL PROTECTED]>wrote: > Hi all, > > I have a specific requirement for query time boosting. > I have to boost a field on the basis of the value returned from one of the > fields of the document. > > Basically, I have the creationDate for a document and in order to introduce > recency factor in the search, i need to give a boost to the creation field, > where the boost value is something like a log(1/x) function and x is the > (presentDate - creationDate). > Till now what I have seen is we can give only a static boost to the > documents. > > In case you can provide a solution to my problem.. please do reply :) > > Thanks a lot, > Regards. > Pooja > -- Regards, Shalin Shekhar Mangar.
Re: Make it more performant - solr 1.3 - 1200msec respond time.
On Thu, Dec 11, 2008 at 5:56 PM, sunnyfr <[EMAIL PROTECTED]> wrote: > > So according to you and everything explained in my post, I did my best to > optimize it ? > Yes it's unique queries. I will try it again and activate cache. > If you run unique queries then it is not a very realistic test. Turn on caching and try running queries from a old access log. > > What you mean by hit the file system? Solr/Lucene has to access the file system to load the results. To avoid this access and processing, there are many caches in Solr. This makes Solr faster. -- Regards, Shalin Shekhar Mangar.
Dynamic Boosting at query time with boost value as another fieldvalue
Hi all, I have a specific requirement for query time boosting. I have to boost a field on the basis of the value returned from one of the fields of the document. Basically, I have the creationDate for a document and in order to introduce recency factor in the search, i need to give a boost to the creation field, where the boost value is something like a log(1/x) function and x is the (presentDate - creationDate). Till now what I have seen is we can give only a static boost to the documents. In case you can provide a solution to my problem.. please do reply :) Thanks a lot, Regards. Pooja
Re: Make it more performant - solr 1.3 - 1200msec respond time.
Hi,
Around 50threads/sec the request bring back " No read Solr server
available" , the gc seems to be quite full, but I didn"t get OOM error,
would love an advice.
Thanks a lot
Details :
8G of memory
4CPU : Intel(R) Xeon(R) CPU5160 @ 3.00GHz
Solr 1.3
# Arguments to pass to the Java virtual machine (JVM).
JAVA_OPTS="-Xms1000m -Xmx4000m -XX:+UseParallelGC
-XX:+HeapDumpOnOutOfMemoryError -Xloggc:gc.log -XX:+PrintGCDetails
-XX:+PrintGCTimeStamps"
5.3G of data with 8,2M of documents.
Thanks a lot for the help
sunnyfr wrote:
>
> Actually I just notices, lot of request didn"t bring back correct answer,
> but " No read Solr server available" so my jmeter didn't take that for an
> error. Obviously out of memory, and a file gc.log is created with :
> 0.054: [GC [PSYoungGen: 5121K->256K(298688K)] 5121K->256K(981376K),
> 0.0020630 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
> 0.056: [Full GC (System) [PSYoungGen: 256K->0K(298688K)] [PSOldGen:
> 0K->180K(682688K)] 256K->180K(981376K) [PSPermGen: 3002K->3002K(21248K)],
> 0.0055170 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
>
> so far my tomcat55 file is configurate like that :
> JAVA_OPTS="-Xms1000m -Xmx4000m -XX:+HeapDumpOnOutOfMemoryError
> -Xloggc:gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps"
>
> My error:
> Dec 11 14:16:27 solr-test jsvc.exec[30653]: Dec 11, 2008 2:16:27 PM
> org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
> path=/admin/ping params={} hits=0 status=0 QTime=1 Dec 11, 2008 2:16:27
> PM org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
> path=/admin/ping params={} status=0 QTime=1 Dec 11, 2008 2:16:27 PM
> org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
> path=/admin/ping params={} hits=0 status=0 QTime=1 Dec 11, 2008 2:16:27
> PM org.apache.solr.core.SolrCore execute INFO: [video] webapp=/solr
> path=/admin/ping params={} status=0 QTime=2
> Dec 11 14:16:27 solr-test jsvc.exec[30653]: java.lang.OutOfMemoryError: GC
> overhead limit exceeded Dumping heap to java_pid30655.hprof ...
>
>
>
>
> Thanks for your help
>
>
>
> Shalin Shekhar Mangar wrote:
>>
>> Are each of those queries unique?
>>
>> First time queries are slower. They are cached by Solr and the same query
>> again will return results very quickly because it won't need to hit the
>> file
>> system.
>>
>> On Thu, Dec 11, 2008 at 4:08 PM, sunnyfr <[EMAIL PROTECTED]> wrote:
>>
>>>
>>> Hi,
>>>
>>> I'm doing a stress test on solr.
>>> I've around 8,5M of doc, the size of my data's directory is 5,6G.
>>>
>>> I've indexed again my data to make it faster, and applied all the last
>>> patch.
>>> My index data store just two field : id and text (which is a copy of
>>> three
>>> fiels)
>>> But I still think it's very long, what do you think?
>>>
>>> For 50request/sec during 40mn, my average respond time : 1235msec.
>>> 49430request.
>>>
>>> When I make this test with 100request second during 10mn and 10 other
>>> minutes with 50 request : my average respond time is 1600msec. Don't you
>>> think it's a bit long.
>>>
>>> Should I partition this index more ? or what should I do to make this
>>> work
>>> faster.
>>> I can read post with people who have just 300msec request for 300Go of
>>> index
>>> partionned ?
>>> My request for collecting all this book is quite complex and have lot of
>>> table linked together, maybe it would be faster if I create a csv file ?
>>>
>>> The server that I'm using for the test has 8G of memory.
>>> 4CPU : Intel(R) Xeon(R) CPU5160 @ 3.00GHz
>>> Tomcat55 : -Xms2000m -Xmx4000m
>>> Solr 1.3.
>>>
>>> What can I modify to make it more performant ? memory, indexation ...?
>>> Does it can come from my request to the mysql database which is too much
>>> linked ?
>>>
>>> Thanks a lot for your help,
>>> Johanna
>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20953079.html
>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
>
--
View this message in context:
http://www.nabble.com/Make-it-more-performant---solr-1.3---1200msec-respond-time.-tp20953079p20955856.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Newbie Question boosting
I can help a bit with <2>... First, keep in mind the difference between index and query time boosting: >From Hossman: ..Index time field bosts are a way to express things like "this documents title is worth twice as much as the title of most documents" query time boosts are a way to express "i care about matches on this clause of my query twice as much as i do about matches to other clauses of my query". Boosting doesn't do anything if your search doesn't use the field you've boosted on, but that's only relevant for index-time boosts. You can't have a query-time boost that doesn't reference the field since that's how you boost at query time by definition. So, say you have documents that you think are more important than run-of-the-mill documents. An example could be an animals database. You might want documents from scholarly sources to have a greater weight, thus be more likely to appear at the top of your search results. These would be fine documents to boost the fields for at index time. That would mean that the scholarly documents would tend to be higher up in your search results. For query time, say you have a genealogy database. Books in this category often have titles like "The Erickson family history in New England, 1700-1930". Now say you have separate fields for title and text. A user searches for "erickson". You want the title to weigh much more than the text since it's much more likely that the user would find that document more valuable than some document about the Swanson family that mentions somewhere in the text that Olaf Swanson married Edna Erickson. So you'd form some query like: title:erickson^10 text:erickson HTH Erick On Thu, Dec 11, 2008 at 1:47 AM, ayyanar <[EMAIL PROTECTED]>wrote: > > I read many articles on boosting still iam not so clear on boosting. Can > anyone explain the following questions with examples? > > 1) Can you given an example for field level boosting and document level > boosting and the difference between two? > > 2) If we set the boost at field level (index time), should the query > contains the that particular field? > For example, if we set the boost for title field, should we create the > termquery for title field? > > Also, based on your experience, can you explain why you need the boosting. > > Thanks, > Ayyanar. A > -- > View this message in context: > http://www.nabble.com/Newbie-Question-boosting-tp20950268p20950268.html > Sent from the Solr - User mailing list archive at Nabble.com. > >
Re: Building Solr from Source
My mistake I saw the maven directories and did not see the build.xml in the src directory so just assumed...My Bad. Anyway built successfully, thanks. Now to apply the field collapsing patch. -John On Dec 11, 2008, at 8:46 AM, Noble Paul നോബിള് नोब्ळ् wrote: Solr uses ant for build install ant On Thu, Dec 11, 2008 at 7:13 PM, John Martyniak <[EMAIL PROTECTED]> wrote: Hi, I have downloaded Maven 2.0.9, and tried to build using "mvn clean install" and "mvn install", nothing works. Can somebody tell me how to build solr from source? I am trying to build the 1.3 source. thank you very much, -John -- --Noble Paul
Re: SolrConfig.xml Replication
Thank you for the quick response. I will keep an eye on that to see how it progresses. On 12/10/08 8:03 PM, "Noble Paul നോബിള് नोब्ळ्" <[EMAIL PROTECTED]> wrote: > This is a known issue and I was planning to take it up soon. > https://issues.apache.org/jira/browse/SOLR-821 > > > On Thu, Dec 11, 2008 at 5:30 AM, Jeff Newburn <[EMAIL PROTECTED]> wrote: >> I am curious as to whether there is a solution to be able to replicate >> solrconfig.xml with the 1.4 replication. The obvious problem is that the >> master would replicate the solrconfig turning all slaves into masters with >> its config. I have also tried on a whim to configure the master and slave >> on the master so that the slave points to the same server but that seems to >> break the replication completely. Please let me know if anybody has any >> ideas >> >> -Jeff >> > >
ruby client - new code base, feedback anyone?
In an attempt to keep my current solr-ruby work manageable, I've created a temp repo on my github account and put everything there: http://github.com/mwmitchell/solr-ruby/tree/master I'll be fleshing out the wiki in the next few days: http://github.com/mwmitchell/solr-ruby/wikis with examples. More tests and rdocs are at the top of the todo. Please feel free to express your ideas and/or opinions about the new code! Matt Mitchell
Re: Upgrade from 1.2 to 1.3 gives 3x slowdown + script!
Yonik >Another thought I just had - do you have autocommit enabled? > No; not as far as I know! The solrconfig.xml from the two versions are equivalent as best I can tell, also they are exactly as provided in the download. The only changes were made by the attached script and should not affect committing. Finally the indexing command has commit=true, which I think means do a single commit at the end of the file? Regards Fergus. >A lucene commit is now more expensive because it syncs the files for >safety. If you commit frequently, this could definitely cause a >slowdown. > >-Yonik > >On Wed, Nov 26, 2008 at 10:54 AM, Fergus McMenemie <[EMAIL PROTECTED]> wrote: >> Hello Grant, >> >> Not much good with Java profilers (yet!) so I thought I >> would send a script! >> >> Details... details! Having decided to produce a script to >> replicate the 1.2 vis 1.3 speed problem. The required rigor >> revealed a lot more. >> >> 1) The faster version I have previously referred to as 1.2, >> was actually a "1.3-dev" I had downloaded as part of the >> solr bootcamp class at ApacheCon Europe 2008. The ID >> string in the CHANGES.txt document is:- >> $Id: CHANGES.txt 643465 2008-04-01 16:10:19Z gsingers $ >> >> 2) I did actually download and speed test a version of 1.2 >> from the internet. It's CHANGES.txt id is:- >> $Id: CHANGES.txt 543263 2007-05-31 21:19:02Z yonik $ >> Speed wise it was about the same as 1.3 at 64min. It also >> had lots of char set issues and is ignored from now on. >> >> 3) The version I was planning to use, till I found this, >> speed issue was the "latest" official version:- >> $Id: CHANGES.txt 694377 2008-09-11 17:40:11Z klaas $ >> I also verified the behavior with a nightly build. >> $Id: CHANGES.txt 712457 2008-11-09 01:24:11Z koji $ >> >> Anyway, The following script indexes the content in 22min >> for the 1.3-dev version and takes 68min for the newer releases >> of 1.3. I took the conf directory from the 1.3dev (bootcamp) >> release and used it replace the conf directory from the >> official 1.3 release. The 3x slow down was still there; it is >> not a configuration issue! >> = >> >> >> >> >> >> >> #! /bin/bash >> >> # This script assumes a /usr/local/tomcat link to whatever version >> # of tomcat you have installed. I have "apache-tomcat-5.5.20" Also >> # /usr/local/tomcat/conf/Catalina/localhost contains no solr.xml. >> # All the following was done as root. >> >> >> # I have a directory /usr/local/ts which contains four versions of solr. The >> # "official" 1.2 along with two 1.3 releases and a version of 1.2 or a >> 1.3beata >> # I got while attending a solr bootcamp. I indexed the same content using the >> # different versions of solr as follows: >> cd /usr/local/ts >> if [ "" ] >> then >> echo "Starting from a-fresh" >> sleep 5 # allow time for me to interrupt! >> cp -Rp apache-solr-bc/example/solr ./solrbc #bc = bootcamp >> cp -Rp apache-solr-nightly/example/solr ./solrnightly >> cp -Rp apache-solr-1.3.0/example/solr ./solr13 >> >> # the gaz is regularly updated and its name keeps changing :-) The page >> # http://earth-info.nga.mil/gns/html/namefiles.htm has a link to the latest >> # version. >> curl >> "http://earth-info.nga.mil/gns/html/geonames_dd_dms_date_20081118.zip"; > >> geonames.zip >> unzip -q geonames.zip >> # delete corrupt blips! >> perl -i -n -e 'print unless >> ($. > 2128495 and $. < 2128505) or >> ($. > 5944254 and $. < 5944260) >> ;' geonames_dd_dms_date_20081118.txt >> #following was used to detect bad short records >> #perl -a -F\\t -n -e ' print "line $. is bad with ",scalar(@F)," args\n" >> if (@F != 26);' geonames_dd_dms_date_20081118.txt >> >> # my set of fields and copyfields for the schema.xml >> fields=' >> >> > required="true" /> >> > stored="true"/> >> > stored="true"/> >> > stored="true"/> >> > stored="true"/> >> > stored="true"/> >> > stored="true"/> >> > stored="true"/> >> > stored="true"/> >> > stored="true"/> >> >> >> ' >> copyfields=' >> >> >> >> ' >> >> # add in my fields and copyfields >> perl -i -p -e "print qq($fields) if s///;" >> solr*/conf/schema.xml >> perl -i -p -e "print qq($copyfields) if s[][];" >> solr*/conf/schema.xml >> # change the unique key and mark the "id" field as not required >> perl -i -p -e "s/id/UNI/i;" >> solr*/conf/schema.xml >> perl -i -p -e 's/required="true"//i if m/> solr*/conf/schema.xml >> # enable remote streaming in solrconfig file >> perl -i -p -e >> 's/enableRemoteStreaming="false"/enableRemoteStreaming="true"/;' >> solr*/conf/solrconfig.xml >> fi >> >> # some constants to keep the curl command shorter >> skip="MODIFY_DATE,RC,UFI,DMS_LAT,DMS_LONG,FC,PC,ADM1,ADM2,POP,ELEV,CC2,NT,LC,SHORT_FORM,GENERIC,SORT_NAME" >> file=`pwd`"/geonames.txt" >> >> export JAVA_OPTS="
Dismax Minimum Match/Stopwords Bug
I have discovered some weirdness with our Minimum Match functionality. Essentially it comes up with absolutely no results on certain queries. Basically, searches with 2 words and 1 being ³the² don¹t have a return result. From what we can gather the minimum match criteria is making it such that if there are 2 words then both are required. Unfortunately, the stopwords are pulled resulting in ³the² being removed and then solr is requiring 2 words when only 1 exists to match on. Is there a way around this? I really need it to either require only non-stopwords or not filter out stopwords. We know stopwords are causing the issue because taking out the stopwords fixes the problem. Also, we can change mm setting to 75% and fix the problem. Example: Brand: The North Face Search: the north (returns no results) Our config is basically: MM: str name="mm">2<-1 FieldType:
Exception running Solr in Weblogic
Guys, I keep getting this exception from time to time on my application when it comunicates with Solr. Does anyone knows if Solr tries to write headers after the response has been sent? <[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <> <1229015137499> <[EMAIL PROTECTED] - appName: 'SolrRepo-cluster1', name: 'solr', context-path: '/solr'] Servlet failed with Exception java.lang.IllegalStateException: Response already committed at weblogic.servlet.internal.ServletResponseImpl.objectIfCommitted(ServletResponseImpl.java:1486) at weblogic.servlet.internal.ServletResponseImpl.sendError(ServletResponseImpl.java:603) at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:362) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:320) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) -- Alexander Ramos Jardim
Re: Exception running Solr in Weblogic
I think somebody just put up a page about Solr and WebLogic up on the Solr Wiki... Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Alexander Ramos Jardim > To: solr-user@lucene.apache.org > Sent: Thursday, December 11, 2008 1:01:16 PM > Subject: Exception running Solr in Weblogic > > Guys, > > I keep getting this exception from time to time on my application when it > comunicates with Solr. Does anyone knows if Solr tries to write headers > after the response has been sent? > > > <[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default > (self-tuning)'> <> <> <> <1229015137499> > <[weblogic.servlet.internal.webappservletcont...@2de805b - appName: > 'SolrRepo-cluster1', name: 'solr', context-path: '/solr'] Servlet failed > with Exception > java.lang.IllegalStateException: Response already committed > at > weblogic.servlet.internal.ServletResponseImpl.objectIfCommitted(ServletResponseImpl.java:1486) > > at > weblogic.servlet.internal.ServletResponseImpl.sendError(ServletResponseImpl.java:603) > > at > org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:362) > > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:320) > > at > weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) > > -- > Alexander Ramos Jardim
Re: Exception running Solr in Weblogic
Can't find it on the wiki. Could you put the url here? 2008/12/11 Otis Gospodnetic > I think somebody just put up a page about Solr and WebLogic up on the Solr > Wiki... > > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message > > From: Alexander Ramos Jardim > > To: solr-user@lucene.apache.org > > Sent: Thursday, December 11, 2008 1:01:16 PM > > Subject: Exception running Solr in Weblogic > > > > Guys, > > > > I keep getting this exception from time to time on my application when it > > comunicates with Solr. Does anyone knows if Solr tries to write headers > > after the response has been sent? > > > > > > <[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default > > (self-tuning)'> <> <> <> <1229015137499> > > <[weblogic.servlet.internal.webappservletcont...@2de805b - appName: > > 'SolrRepo-cluster1', name: 'solr', context-path: '/solr'] Servlet failed > > with Exception > > java.lang.IllegalStateException: Response already committed > > at > > > weblogic.servlet.internal.ServletResponseImpl.objectIfCommitted(ServletResponseImpl.java:1486) > > > > at > > > weblogic.servlet.internal.ServletResponseImpl.sendError(ServletResponseImpl.java:603) > > > > at > > > org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:362) > > > > at > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:320) > > > > at > > > weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) > > > > -- > > Alexander Ramos Jardim > > -- Alexander Ramos Jardim
Re: Taxonomy Support on Solr
This is what Hoss was hinting at yesterday (or was that on the Lucene list?). You can do that if you encode the hierarchy in a field properly., e.g. "/A /B /1" may be one doc's field. "/A /B /2" may be another doc's field. THen you just have to figure out how to query that to get a sub-tree. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: "Jana, Kumar Raja" > To: solr-user@lucene.apache.org > Sent: Thursday, December 11, 2008 5:03:02 AM > Subject: Taxonomy Support on Solr > > Hi, > > Any plans of supporting user-defined classifications on Solr? Is there > any component which returns all the children of a node (till the leaf > node) when I search for any node? > > May be this would help: > > Say I have a few SolrDocuments classified as: > > A > B--C > 123 8--9 > > (I.e A has 2 child nodes B and C. B has 3 child nodes 1,2,3 and C has 2 > child nodes 8,9) > When my search criteria matches B, my results should contain B as well > as 1,2 and 3 too. > Search for A would return all the nodes mentioned above. > > -Kumar
Applying Field Collapsing Patch
Hi, I am trying to apply Ivan's field collapsing patch to solr 1.3 (not a nightly), and it continously fails. I am using the following command: patch -p0 -i collapsing-patch-to-1.3.0-ivan_2.patch --dry-run I am in the apache-solr directory, and have read write for all files directories and files. I am get the following results: patching file src/test/org/apache/solr/search/TestDocSet.java Hunk #1 FAILED at 88. 1 out of 1 hunk FAILED -- saving rejects to file src/test/org/apache/ solr/search/TestDocSet.java.rej patching file src/java/org/apache/solr/search/CollapseFilter.java patching file src/java/org/apache/solr/search/DocSet.java Hunk #1 FAILED at 195. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/apache/ solr/search/DocSet.java.rej patching file src/java/org/apache/solr/search/NegatedDocSet.java patching file src/java/org/apache/solr/search/SolrIndexSearcher.java Hunk #1 FAILED at 1357. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/apache/ solr/search/SolrIndexSearcher.java.rej patching file src/java/org/apache/solr/common/params/CollapseParams.java patching file src/java/org/apache/solr/handler/component/ CollapseComponent.java Also the '.rej' files are not created. Does anybody have any ideas? thanks in advance for the help. -John
Re: Taxonomy Support on Solr
I use this workaround all the time. When I need to put the hierarchy which a product belongs, I simply arranje all the nodes as: "a ^ b ^ c ^ d" 2008/12/11 Otis Gospodnetic > This is what Hoss was hinting at yesterday (or was that on the Lucene > list?). You can do that if you encode the hierarchy in a field properly., > e.g. "/A /B /1" may be one doc's field. "/A /B /2" may be another doc's > field. THen you just have to figure out how to query that to get a > sub-tree. > > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > - Original Message > > From: "Jana, Kumar Raja" > > To: solr-user@lucene.apache.org > > Sent: Thursday, December 11, 2008 5:03:02 AM > > Subject: Taxonomy Support on Solr > > > > Hi, > > > > Any plans of supporting user-defined classifications on Solr? Is there > > any component which returns all the children of a node (till the leaf > > node) when I search for any node? > > > > May be this would help: > > > > Say I have a few SolrDocuments classified as: > > > > A > > B--C > > 123 8--9 > > > > (I.e A has 2 child nodes B and C. B has 3 child nodes 1,2,3 and C has 2 > > child nodes 8,9) > > When my search criteria matches B, my results should contain B as well > > as 1,2 and 3 too. > > Search for A would return all the nodes mentioned above. > > > > -Kumar > > -- Alexander Ramos Jardim
exceeded limit of maxWarmingSearchers=4
We’re using Solr as a backend indexer/search engine to support an AJAX based consumer application. Basically, when users of our system create “Documents” in our product, we commit right away, because we want to immediately re-query and get counts back from Solr to update the user’s interface. Occasionally, usually under load, we see the following error... ERROR [IndexSubscription] error committing changes to solr java.lang.RuntimeException: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=4, try again later. My understanding of Solr’s caches and warming searches is that we get nearly no value out of them because with many users, each only ever querying their own data, the caches would quickly be marked irrelevant by new unique user queries. So, we’ve tried setting true And we’ve set all the cache autowarmCounts=”0”, for example: Yet the same problem is periodically logged. Is there a way to disable warming Searches? Should we even be trying this? Is there a better configuration, i.e. Do we just set Warming Searches to a higher number? I read in solr...@jira that removing this configuration all together forces the number of warming Searches be infinite – that can’t be good? _ Send e-mail anywhere. No map, no compass. http://windowslive.com/Explore/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008
Re: Taxonomy Support on Solr
I had a similar problem and I solved it by making the directory a multi-valued field in the index and giving each directory a unique id. So for example, a document in directory 2 would contain in the index: "dir_id:A dir_id:B dir_id:2". A search on any of those fields will then return directory 2. Conversely, a search for dir_id:2 will not return directory B. I hope I understood your question correctly. Mark Ferguson On Thu, Dec 11, 2008 at 3:03 AM, Jana, Kumar Raja wrote: > Hi, > > Any plans of supporting user-defined classifications on Solr? Is there > any component which returns all the children of a node (till the leaf > node) when I search for any node? > > May be this would help: > > Say I have a few SolrDocuments classified as: > >A >B--C >123 8--9 > > (I.e A has 2 child nodes B and C. B has 3 child nodes 1,2,3 and C has 2 > child nodes 8,9) > When my search criteria matches B, my results should contain B as well > as 1,2 and 3 too. > Search for A would return all the nodes mentioned above. > > -Kumar > > >
Re: exceeded limit of maxWarmingSearchers=4
chip correra wrote: We’re using Solr as a backend indexer/search engine to support an AJAX based consumer application. Basically, when users of our system create “Documents” in our product, we commit right away, because we want to immediately re-query and get counts back from Solr to update the user’s interface. Occasionally, usually under load, we see the following error... ERROR [IndexSubscription] error committing changes to solr java.lang.RuntimeException: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=4, try again later. My understanding of Solr’s caches and warming searches is that we get nearly no value out of them because with many users, each only ever querying their own data, the caches would quickly be marked irrelevant by new unique user queries. So, we’ve tried setting true And we’ve set all the cache autowarmCounts=”0”, for example: Yet the same problem is periodically logged. Is there a way to disable warming Searches? Should we even be trying this? Is there a better configuration, i.e. Do we just set Warming Searches to a higher number? I read in solr...@jira that removing this configuration all together forces the number of warming Searches be infinite – that can’t be good? _ Send e-mail anywhere. No map, no compass. http://windowslive.com/Explore/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008 It likely means your committing too often. Pretty often if it happens even with a cold searcher. Whats your commit policy?
Re: Applying Field Collapsing Patch
Are you sure you have a clean copy of the source? Every time I've applied his patch I grab a fresh copy of the tarball and run the exact same command, it always works for me. Now, whether the collapsing actually works is a different matter... -- Steve On Dec 11, 2008, at 1:29 PM, John Martyniak wrote: Hi, I am trying to apply Ivan's field collapsing patch to solr 1.3 (not a nightly), and it continously fails. I am using the following command: patch -p0 -i collapsing-patch-to-1.3.0-ivan_2.patch --dry-run I am in the apache-solr directory, and have read write for all files directories and files. I am get the following results: patching file src/test/org/apache/solr/search/TestDocSet.java Hunk #1 FAILED at 88. 1 out of 1 hunk FAILED -- saving rejects to file src/test/org/apache/ solr/search/TestDocSet.java.rej patching file src/java/org/apache/solr/search/CollapseFilter.java patching file src/java/org/apache/solr/search/DocSet.java Hunk #1 FAILED at 195. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/apache/ solr/search/DocSet.java.rej patching file src/java/org/apache/solr/search/NegatedDocSet.java patching file src/java/org/apache/solr/search/SolrIndexSearcher.java Hunk #1 FAILED at 1357. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/apache/ solr/search/SolrIndexSearcher.java.rej patching file src/java/org/apache/solr/common/params/ CollapseParams.java patching file src/java/org/apache/solr/handler/component/ CollapseComponent.java Also the '.rej' files are not created. Does anybody have any ideas? thanks in advance for the help. -John
RE: exceeded limit of maxWarmingSearchers=4
We commit immediately after each and every document submit. I think we have to because we want to immediately retrieve a count on the number of documents of that type, including the one that we just submitted. And my understanding is that if we don't commit immediately, the new document will not be part of the query results. > Date: Thu, 11 Dec 2008 14:09:47 -0500 > From: markrmil...@gmail.com > To: solr-user@lucene.apache.org > Subject: Re: exceeded limit of maxWarmingSearchers=4 > > chip correra wrote: > > We’re using Solr as a backend indexer/search engine to support an > > AJAX based consumer application. Basically, when users of our system > > create “Documents” in our product, we commit right away, because we want to > > immediately re-query and get counts back from Solr to update the user’s > > interface. Occasionally, usually under load, we see the following error... > > > > ERROR [IndexSubscription] error committing changes to solr > > java.lang.RuntimeException: org.apache.solr.common.SolrException: Error > > opening new searcher. exceeded limit of maxWarmingSearchers=4, try again > > later. > > > > My understanding of Solr’s caches and warming searches is that we get > > nearly no value out of them because with many users, each only ever > > querying their own data, the caches would quickly be marked irrelevant by > > new unique user queries. > > > > So, we’ve tried setting true > > And we’ve set all the cache autowarmCounts=”0”, for example: > class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0"/> > > > > Yet the same problem is periodically logged. > > > > Is there a way to disable warming Searches? > > > > Should we even be trying this? > > > > Is there a better configuration, i.e. Do we just set Warming Searches to a > > higher number? > > > > I read in solr...@jira that removing this configuration all together forces > > the number of warming Searches be infinite – that can’t be good? > > > > > > _ > > Send e-mail anywhere. No map, no compass. > > http://windowslive.com/Explore/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008 > > > It likely means your committing too often. Pretty often if it happens > even with a cold searcher. Whats your commit policy? > _ Send e-mail anywhere. No map, no compass. http://windowslive.com/Explore/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_122008
Re: Exception running Solr in Weblogic
http://wiki.apache.org/solr/SolrWebSphere On Fri, Dec 12, 2008 at 12:00 AM, Alexander Ramos Jardim < alexander.ramos.jar...@gmail.com> wrote: > Can't find it on the wiki. Could you put the url here? > > 2008/12/11 Otis Gospodnetic > > > I think somebody just put up a page about Solr and WebLogic up on the > Solr > > Wiki... > > > > > > Otis > > -- > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > > > > > - Original Message > > > From: Alexander Ramos Jardim > > > To: solr-user@lucene.apache.org > > > Sent: Thursday, December 11, 2008 1:01:16 PM > > > Subject: Exception running Solr in Weblogic > > > > > > Guys, > > > > > > I keep getting this exception from time to time on my application when > it > > > comunicates with Solr. Does anyone knows if Solr tries to write headers > > > after the response has been sent? > > > > > > > > > <[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default > > > (self-tuning)'> <> <> <> <1229015137499> > > > <[weblogic.servlet.internal.webappservletcont...@2de805b - appName: > > > 'SolrRepo-cluster1', name: 'solr', context-path: '/solr'] Servlet > failed > > > with Exception > > > java.lang.IllegalStateException: Response already committed > > > at > > > > > > weblogic.servlet.internal.ServletResponseImpl.objectIfCommitted(ServletResponseImpl.java:1486) > > > > > > at > > > > > > weblogic.servlet.internal.ServletResponseImpl.sendError(ServletResponseImpl.java:603) > > > > > > at > > > > > > org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:362) > > > > > > at > > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:320) > > > > > > at > > > > > > weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) > > > > > > -- > > > Alexander Ramos Jardim > > > > > > > -- > Alexander Ramos Jardim > -- Regards, Shalin Shekhar Mangar.
Re: Problems with SOLR-236 (field collapsing)
Thanks Doug, removing "query" definitely helped. I just switched to Ivan's new patch (which definitely helped a lot - no SEVERE errors now - thanks Ivan!) but I'm still struggling with faceting myself. Basically, I can tell that faceting is happening after the collapse - because the facet counts are definitely lower than they would be otherwise. For example, with one search, I'd have 196 results with no collapsing, I get 120 results with collapsing - but the facet count is 119??? In other searches the difference is more drastic - In another search, I get 61 results without collapsing, 61 with collapsing, but the facet count is 39. Looking at it for a while now, I think I can guess what the problem might be... The incorrect counts seem to only happen when the term in question does not occur evenly across all duplicates of a document. That is, multiple document records may exist for the same image (it's an image search engine), but each document will have different terms in different fields depending on the audience it's targeting. So, when you collapse, the counts are lower than they should be because when you actually execute a search with that facet's term included in the query, *all* the documents after collapsing will be ones that have that term. Here's an illustration: Collapse field is "link_id", facet field is "keyword": Doc 1: id: 123456, link_id: 2, keyword: Black, Printed, Dress Doc 2: id: 123457, link_id: 2, keyword: Black, Shoes, Patent Doc 3: id: 123458, link_id: 2, keyword: Red, Hat, Felt Doc 4: id: 123459, link_id:1, keyword: Felt, Hat, Black So, when you collapse, only two of these documents are in the result set (123456, 123459), and only the keywords Black, Printed, Dress, Felt, and Hat are counted. The facet count for Black is 2, the facet count for Felt is 1. If you choose Black and add it to your query, you get 2 results (great). However, if you add *Felt* to your query, you get 2 results (because a different document for link_id 2 is chosen in that query than is in the more general query from which the facets are produced). I think what needs to happen here is that all the terms for all the documents that are collapsed together need to be included (just once) with the document that gets counted for faceting. In this example, when the document for link_id 2 is counted, it would need to appear to the facet counter to have keywords Black, Printed, Dress, Shoes, Patent, Red, Hat, and Felt, as opposed to just Black, Printed, and Dress. Unfortunately, not knowing Java at all really, I have absolutely no idea how this change would be implemented... I mean I can tweak here or there, but I think this is above my pay grade. I've looked at the code affected by the patch and the code for faceting but I can't make heads or tails of it. I think I'll go post this over on the JIRA... Any ideas? -- Steve On Dec 10, 2008, at 8:52 AM, Doug Steigerwald wrote: The first output is from the query component. You might just need to make the collapse component first and remove the query component completely. We perform geographic searching with localsolr first (if we need to), and then try to collapse those results (if collapse=true). If we don't have any results yet, that's the only time we use the standard query component. I'm making sure we set the builder.setNeedDocSet=false and then I modified the query component to only execute when builder.isNeedDocSet=true. In the field collapsing patch that I'm using, I've got code to remove a previous 'response' from the builder.rsp so we don't have duplicates. Now, if I could get field collapsing to work properly with a docSet/ docList from localsolr and also have faceting work, I'd be golden. Doug On Dec 9, 2008, at 9:37 PM, Stephen Weiss wrote: Hi Tracy, Well, I managed to get it working (I think) but the weird thing is, in the XML output it gives both recordsets (the filtered and unfiltered - filtered second). In the JSON (the one I actually use anyway, at least) I only get the filtered results (as expected). In my core's solrconfig.xml, I added: class="org.apache.solr.handler.component.CollapseComponent" /> (I'm not sure if it's supposed to go anywhere in particular but for me it's right before StandardRequestHandler) and then within StandardRequestHandler: explicit query facet mlt highlight debug collapse Which is basically all the default values plus collapse. Not sure if this was needed for prior versions, I don't see it in any patch files (I just got a vague idea from looking at a comment from someone else who said it wasn't working for them). It would kinda be nice if someone working on the code might throw us a bone and say explicitly what the right options to put in the config file are (if there are even supposed to be
Re: exceeded limit of maxWarmingSearchers=4
It sounds like you need real-time search, where documents are available in the next query. Solr doesn't do that. That is a pretty rare feature and must be designed in at the start. The usual workaround is to have a main index plus a small delta index and search both. Deletes have to be handled separately. The Sphinx docs have a description of how to do that with their engine: http://www.sphinxsearch.com/docs/current.html#live-updates wunder On 12/11/08 10:44 AM, "chip correra" wrote: > > > We¹re using Solr as a backend indexer/search engine to support an AJAX > based consumer application. Basically, when users of our system create > ³Documents² in our product, we commit right away, because we want to > immediately re-query and get counts back from Solr to update the user¹s > interface. Occasionally, usually under load, we see the following error... > > ERROR [IndexSubscription] error committing changes to solr > java.lang.RuntimeException: org.apache.solr.common.SolrException: Error > opening new searcher. exceeded limit of maxWarmingSearchers=4, try again > later. > > My understanding of Solr¹s caches and warming searches is that we get nearly > no value out of them because with many users, each only ever querying their > own data, the caches would quickly be marked irrelevant by new unique user > queries. > > So, we¹ve tried setting true > And we¹ve set all the cache autowarmCounts=²0², for example: class="solr.LRUCache" size="512" initialSize="512" autowarmCount="0"/> > > Yet the same problem is periodically logged. > > Is there a way to disable warming Searches? > > Should we even be trying this? > > Is there a better configuration, i.e. Do we just set Warming Searches to a > higher number? > > I read in solr...@jira that removing this configuration all together forces > the number of warming Searches be infinite that can¹t be good? > > > _ > Send e-mail anywhere. No map, no compass. > http://windowslive.com/Explore/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_ > 122008
Re: Applying Field Collapsing Patch
thanks for the advice. I just downloaded a completely clean version, haven't even tried to build it yet. Applied the same, and I received exactly the same results. Do you only apply the ivan patch 2? What version of patch are you running? -John On Dec 11, 2008, at 2:10 PM, Stephen Weiss wrote: Are you sure you have a clean copy of the source? Every time I've applied his patch I grab a fresh copy of the tarball and run the exact same command, it always works for me. Now, whether the collapsing actually works is a different matter... -- Steve On Dec 11, 2008, at 1:29 PM, John Martyniak wrote: Hi, I am trying to apply Ivan's field collapsing patch to solr 1.3 (not a nightly), and it continously fails. I am using the following command: patch -p0 -i collapsing-patch-to-1.3.0-ivan_2.patch --dry-run I am in the apache-solr directory, and have read write for all files directories and files. I am get the following results: patching file src/test/org/apache/solr/search/TestDocSet.java Hunk #1 FAILED at 88. 1 out of 1 hunk FAILED -- saving rejects to file src/test/org/ apache/solr/search/TestDocSet.java.rej patching file src/java/org/apache/solr/search/CollapseFilter.java patching file src/java/org/apache/solr/search/DocSet.java Hunk #1 FAILED at 195. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/ apache/solr/search/DocSet.java.rej patching file src/java/org/apache/solr/search/NegatedDocSet.java patching file src/java/org/apache/solr/search/SolrIndexSearcher.java Hunk #1 FAILED at 1357. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/ apache/solr/search/SolrIndexSearcher.java.rej patching file src/java/org/apache/solr/common/params/ CollapseParams.java patching file src/java/org/apache/solr/handler/component/ CollapseComponent.java Also the '.rej' files are not created. Does anybody have any ideas? thanks in advance for the help. -John
Re: exceeded limit of maxWarmingSearchers=4
Also, if you are using solr 1.3, solr 1.4 will reopen readers rather than open them again. This means only changed segments have to be reloaded. If you turn off all the caches and use a bit higher merge factor, maybe a low max merge docs, you can prob get things a lot quicker. There will still be instances where it can't keep up I'm sure though. In that case, just ignore the warning. You might set the limit lower even. Some stale views will go out, but it should be faster than what you are getting now (and your getting the stale views now, as walter says, solr doesnt do realtime indexing). Walter Underwood wrote: It sounds like you need real-time search, where documents are available in the next query. Solr doesn't do that. That is a pretty rare feature and must be designed in at the start. The usual workaround is to have a main index plus a small delta index and search both. Deletes have to be handled separately. The Sphinx docs have a description of how to do that with their engine: http://www.sphinxsearch.com/docs/current.html#live-updates wunder On 12/11/08 10:44 AM, "chip correra" wrote: We¹re using Solr as a backend indexer/search engine to support an AJAX based consumer application. Basically, when users of our system create ³Documents² in our product, we commit right away, because we want to immediately re-query and get counts back from Solr to update the user¹s interface. Occasionally, usually under load, we see the following error... ERROR [IndexSubscription] error committing changes to solr java.lang.RuntimeException: org.apache.solr.common.SolrException: Error opening new searcher. exceeded limit of maxWarmingSearchers=4, try again later. My understanding of Solr¹s caches and warming searches is that we get nearly no value out of them because with many users, each only ever querying their own data, the caches would quickly be marked irrelevant by new unique user queries. So, we¹ve tried setting true And we¹ve set all the cache autowarmCounts=²0², for example: Yet the same problem is periodically logged. Is there a way to disable warming Searches? Should we even be trying this? Is there a better configuration, i.e. Do we just set Warming Searches to a higher number? I read in solr...@jira that removing this configuration all together forces the number of warming Searches be infinite that can¹t be good? _ Send e-mail anywhere. No map, no compass. http://windowslive.com/Explore/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_anywhere_ 122008
Re: Applying Field Collapsing Patch
Yes, only ivan patch 2 (and before, only ivan patch 1), my sense was these patches were meant to be used in isolation (there were no notes saying to apply any other patches first). Are you using patches for any other purpose (non-SOLR-236)? Maybe you need to apply this one first, then those patches. For me using any patch makes me nervous (we have a pretty strict policy about using beta code anywhere), I'm only doing it this once because it's absolutely necessary to provide the functionality desired. -- Steve On Dec 11, 2008, at 2:53 PM, John Martyniak wrote: thanks for the advice. I just downloaded a completely clean version, haven't even tried to build it yet. Applied the same, and I received exactly the same results. Do you only apply the ivan patch 2? What version of patch are you running? -John On Dec 11, 2008, at 2:10 PM, Stephen Weiss wrote: Are you sure you have a clean copy of the source? Every time I've applied his patch I grab a fresh copy of the tarball and run the exact same command, it always works for me. Now, whether the collapsing actually works is a different matter... -- Steve On Dec 11, 2008, at 1:29 PM, John Martyniak wrote: Hi, I am trying to apply Ivan's field collapsing patch to solr 1.3 (not a nightly), and it continously fails. I am using the following command: patch -p0 -i collapsing-patch-to-1.3.0-ivan_2.patch --dry-run I am in the apache-solr directory, and have read write for all files directories and files. I am get the following results: patching file src/test/org/apache/solr/search/TestDocSet.java Hunk #1 FAILED at 88. 1 out of 1 hunk FAILED -- saving rejects to file src/test/org/ apache/solr/search/TestDocSet.java.rej patching file src/java/org/apache/solr/search/CollapseFilter.java patching file src/java/org/apache/solr/search/DocSet.java Hunk #1 FAILED at 195. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/ apache/solr/search/DocSet.java.rej patching file src/java/org/apache/solr/search/NegatedDocSet.java patching file src/java/org/apache/solr/search/SolrIndexSearcher.java Hunk #1 FAILED at 1357. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/ apache/solr/search/SolrIndexSearcher.java.rej patching file src/java/org/apache/solr/common/params/ CollapseParams.java patching file src/java/org/apache/solr/handler/component/ CollapseComponent.java Also the '.rej' files are not created. Does anybody have any ideas? thanks in advance for the help. -John
move /solr directory from /tomcat/bin/
Hey there, I would like to change the default directory where solr looks for the config files and index. Let's say I would like to put: /opt/tomcat/bin/solr/data/index in /var/searchengine_data/index and /opt/tomcat/bin/solr/conf in /usr/home/searchengine_files/conf Is there any way to do it via configuration or I should modify the SolrResourceLoader? Thanks in advance -- View this message in context: http://www.nabble.com/move--solr-directory-from--tomcat-bin--tp20963811p20963811.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: move /solr directory from /tomcat/bin/
You can set the home directory in your Tomcat context snippet/file. http://wiki.apache.org/solr/SolrTomcat#head-7036378fa48b79c0797cc8230a8a a0965412fb2e This controls where Solr looks for solrconfig.xml and schema.xml. The solrconfig.xml in turn specifies where to find the data directory. -Original Message- From: Marc Sturlese [mailto:marc.sturl...@gmail.com] Sent: Thursday, December 11, 2008 12:20 PM To: solr-user@lucene.apache.org Subject: move /solr directory from /tomcat/bin/ Hey there, I would like to change the default directory where solr looks for the config files and index. Let's say I would like to put: /opt/tomcat/bin/solr/data/index in /var/searchengine_data/index and /opt/tomcat/bin/solr/conf in /usr/home/searchengine_files/conf Is there any way to do it via configuration or I should modify the SolrResourceLoader? Thanks in advance -- View this message in context: http://www.nabble.com/move--solr-directory-from--tomcat-bin--tp20963811p 20963811.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Applying Field Collapsing Patch
It was a completely clean install. I downloaded it from one of mirrors right before applying the patch to it. Very troubling. Any other suggestions or ideas? I am running it on Mac OS Maybe I will try looking for some answers around that. -John On Dec 11, 2008, at 3:05 PM, Stephen Weiss wrote: Yes, only ivan patch 2 (and before, only ivan patch 1), my sense was these patches were meant to be used in isolation (there were no notes saying to apply any other patches first). Are you using patches for any other purpose (non-SOLR-236)? Maybe you need to apply this one first, then those patches. For me using any patch makes me nervous (we have a pretty strict policy about using beta code anywhere), I'm only doing it this once because it's absolutely necessary to provide the functionality desired. -- Steve On Dec 11, 2008, at 2:53 PM, John Martyniak wrote: thanks for the advice. I just downloaded a completely clean version, haven't even tried to build it yet. Applied the same, and I received exactly the same results. Do you only apply the ivan patch 2? What version of patch are you running? -John On Dec 11, 2008, at 2:10 PM, Stephen Weiss wrote: Are you sure you have a clean copy of the source? Every time I've applied his patch I grab a fresh copy of the tarball and run the exact same command, it always works for me. Now, whether the collapsing actually works is a different matter... -- Steve On Dec 11, 2008, at 1:29 PM, John Martyniak wrote: Hi, I am trying to apply Ivan's field collapsing patch to solr 1.3 (not a nightly), and it continously fails. I am using the following command: patch -p0 -i collapsing-patch-to-1.3.0-ivan_2.patch --dry-run I am in the apache-solr directory, and have read write for all files directories and files. I am get the following results: patching file src/test/org/apache/solr/search/TestDocSet.java Hunk #1 FAILED at 88. 1 out of 1 hunk FAILED -- saving rejects to file src/test/org/ apache/solr/search/TestDocSet.java.rej patching file src/java/org/apache/solr/search/CollapseFilter.java patching file src/java/org/apache/solr/search/DocSet.java Hunk #1 FAILED at 195. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/ apache/solr/search/DocSet.java.rej patching file src/java/org/apache/solr/search/NegatedDocSet.java patching file src/java/org/apache/solr/search/ SolrIndexSearcher.java Hunk #1 FAILED at 1357. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/ apache/solr/search/SolrIndexSearcher.java.rej patching file src/java/org/apache/solr/common/params/ CollapseParams.java patching file src/java/org/apache/solr/handler/component/ CollapseComponent.java Also the '.rej' files are not created. Does anybody have any ideas? thanks in advance for the help. -John
Request for paid help...
We are currently using Apache, SOLR and Java to populate the lucene/solr index. We need someone to upgrade our SOLR to the newest version, review our schema (solr) and solr config to performance tune and make suggestions. Also the schema and solr config most likely contain information we don't need as default examples. This is actually on a very powerful Windows2k3 dedicated server. Current version : Apache 2.2.8 Mysql 5x Solr Specification Version: 1.2.0 Solr Implementation Version: 1.2.0 - Yonik - 2007-06-02 17:35:12 Lucene Specification Version: 2007-05-20_00-04-53 Lucene Implementation Version: build 2007-05-20 I am willing to pay up to $250. -Craig cstadle...@hotmail.com
Re: Exception running Solr in Weblogic
http://wiki.apache.org/solr/SolrWebSphere Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: Alexander Ramos Jardim > To: solr-user@lucene.apache.org > Sent: Thursday, December 11, 2008 1:30:03 PM > Subject: Re: Exception running Solr in Weblogic > > Can't find it on the wiki. Could you put the url here? > > 2008/12/11 Otis Gospodnetic > > > I think somebody just put up a page about Solr and WebLogic up on the Solr > > Wiki... > > > > > > Otis > > -- > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > > > > > - Original Message > > > From: Alexander Ramos Jardim > > > To: solr-user@lucene.apache.org > > > Sent: Thursday, December 11, 2008 1:01:16 PM > > > Subject: Exception running Solr in Weblogic > > > > > > Guys, > > > > > > I keep getting this exception from time to time on my application when it > > > comunicates with Solr. Does anyone knows if Solr tries to write headers > > > after the response has been sent? > > > > > > > > > <[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default > > > (self-tuning)'> <> <> <> <1229015137499> > > > <[weblogic.servlet.internal.webappservletcont...@2de805b - appName: > > > 'SolrRepo-cluster1', name: 'solr', context-path: '/solr'] Servlet failed > > > with Exception > > > java.lang.IllegalStateException: Response already committed > > > at > > > > > > weblogic.servlet.internal.ServletResponseImpl.objectIfCommitted(ServletResponseImpl.java:1486) > > > > > > at > > > > > > weblogic.servlet.internal.ServletResponseImpl.sendError(ServletResponseImpl.java:603) > > > > > > at > > > > > > org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:362) > > > > > > at > > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:320) > > > > > > at > > > > > weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:42) > > > > > > -- > > > Alexander Ramos Jardim > > > > > > > -- > Alexander Ramos Jardim
Re: ExtractingRequestHandler and XmlUpdateHandler
On Dec 10, 2008, at 10:21 PM, Jacob Singh wrote: Hey folks, I'm looking at implementing ExtractingRequestHandler in the Apache_Solr_PHP library, and I'm wondering what we can do about adding meta-data. I saw the docs, which suggests you use different post headers to pass field values along with ext.literal. Is there anyway to use the XmlUpdateHandler instead along with a document? I'm not sure how this would work, perhaps it would require 2 trips, perhaps the XML would be in the post "content" and the file in something else? The thing is we would need to refactor the class pretty heavily in this case when indexing RichDocs and we were hoping to avoid it. I'm not sure I follow how the XmlUpdateHandler plays in, can you explain a little more? My PHP is weak, but maybe some code will help... Thanks, Jacob -- +1 510 277-0891 (o) +91 33 7458 (m) web: http://pajamadesign.com Skype: pajamadesign Yahoo: jacobsingh AIM: jacobsingh gTalk: jacobsi...@gmail.com
Format of highlighted fields in query response
Hello, I am making a query to my Solr server in which I would like to have a number of fields returned, with highlighting if available. I've noticed that in the query response, I get back both the original field name and then in a different section, the highlighted snippet. I am wondering if there is a parameter which will allow me to collapse this data, returning only the highlighted snippet in the doc itself, when available. For example, I am currently receiving the following data: 0.2963915 Chinese Visa Information http://www.danwei.org/china_information/chinese_visa_information.php 01598a6e06190bd8b05c8b03f51233a1 ... and farther down ... TITLE: Chinese Visa Information I would like it to just look like this: 0.2963915 Chinese Visa Information http://www.danwei.org/china_information/chinese_visa_information.php 01598a6e06190bd8b05c8b03f51233a1 The reason I would prefer this second response format is because I don't need the first field, and it greatly simplifies my call to QueryResponse.getBeans() in SolrJ, as it will fill in everything I need in one call. Thanks very much, Mark Ferguson
Re: Applying Field Collapsing Patch
I am also a Mac user. 10.5.5. I generally compile on OS X then upload to Debian (Debian's java just isn't as friendly to me). Perhaps try this old Mac adage - have you repaired permissions? :-) If you want I can send you a tar ball of the patched code off-list so you can move on. -- Steve On Dec 11, 2008, at 3:50 PM, John Martyniak wrote: It was a completely clean install. I downloaded it from one of mirrors right before applying the patch to it. Very troubling. Any other suggestions or ideas? I am running it on Mac OS Maybe I will try looking for some answers around that. -John On Dec 11, 2008, at 3:05 PM, Stephen Weiss wrote: Yes, only ivan patch 2 (and before, only ivan patch 1), my sense was these patches were meant to be used in isolation (there were no notes saying to apply any other patches first). Are you using patches for any other purpose (non-SOLR-236)? Maybe you need to apply this one first, then those patches. For me using any patch makes me nervous (we have a pretty strict policy about using beta code anywhere), I'm only doing it this once because it's absolutely necessary to provide the functionality desired. -- Steve On Dec 11, 2008, at 2:53 PM, John Martyniak wrote: thanks for the advice. I just downloaded a completely clean version, haven't even tried to build it yet. Applied the same, and I received exactly the same results. Do you only apply the ivan patch 2? What version of patch are you running? -John On Dec 11, 2008, at 2:10 PM, Stephen Weiss wrote: Are you sure you have a clean copy of the source? Every time I've applied his patch I grab a fresh copy of the tarball and run the exact same command, it always works for me. Now, whether the collapsing actually works is a different matter... -- Steve On Dec 11, 2008, at 1:29 PM, John Martyniak wrote: Hi, I am trying to apply Ivan's field collapsing patch to solr 1.3 (not a nightly), and it continously fails. I am using the following command: patch -p0 -i collapsing-patch-to-1.3.0-ivan_2.patch --dry-run I am in the apache-solr directory, and have read write for all files directories and files. I am get the following results: patching file src/test/org/apache/solr/search/TestDocSet.java Hunk #1 FAILED at 88. 1 out of 1 hunk FAILED -- saving rejects to file src/test/org/ apache/solr/search/TestDocSet.java.rej patching file src/java/org/apache/solr/search/CollapseFilter.java patching file src/java/org/apache/solr/search/DocSet.java Hunk #1 FAILED at 195. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/ apache/solr/search/DocSet.java.rej patching file src/java/org/apache/solr/search/NegatedDocSet.java patching file src/java/org/apache/solr/search/ SolrIndexSearcher.java Hunk #1 FAILED at 1357. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/ apache/solr/search/SolrIndexSearcher.java.rej patching file src/java/org/apache/solr/common/params/ CollapseParams.java patching file src/java/org/apache/solr/handler/component/ CollapseComponent.java Also the '.rej' files are not created. Does anybody have any ideas? thanks in advance for the help. -John
RE: Query Performance while updating teh index
We are still having this problem. I am wondering if it can be fixed with autowarm settings. Is there a reliable formula for determining the autowarm settings? -- View this message in context: http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20968516.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Query Performance while updating teh index
Oleg, The reliable formula is situation-specific, I think. One sure way to decrease the warm time is to minimize the number of items to copy from old caches to new caches on warmup. Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch - Original Message > From: oleg_gnatovskiy > To: solr-user@lucene.apache.org > Sent: Thursday, December 11, 2008 8:43:26 PM > Subject: RE: Query Performance while updating teh index > > > We are still having this problem. I am wondering if it can be fixed with > autowarm settings. Is there a reliable formula for determining the autowarm > settings? > -- > View this message in context: > http://www.nabble.com/Query-Performance-while-updating-the-index-tp20452835p20968516.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: Applying Field Collapsing Patch
Have you tried just checking out (or exporting) the source from SVN and applying the patch? Works fine for me that way. $ svn co http://svn.apache.org/repos/asf/lucene/solr/tags/ release-1.3.0 solr-1.3.0 $ cd solr-1.3.0 ; patch -p0 < ~/Downloads/collapsing-patch-to-1.3.0- ivan_2.patch Doug On Dec 11, 2008, at 3:50 PM, John Martyniak wrote: It was a completely clean install. I downloaded it from one of mirrors right before applying the patch to it. Very troubling. Any other suggestions or ideas? I am running it on Mac OS Maybe I will try looking for some answers around that. -John On Dec 11, 2008, at 3:05 PM, Stephen Weiss wrote: Yes, only ivan patch 2 (and before, only ivan patch 1), my sense was these patches were meant to be used in isolation (there were no notes saying to apply any other patches first). Are you using patches for any other purpose (non-SOLR-236)? Maybe you need to apply this one first, then those patches. For me using any patch makes me nervous (we have a pretty strict policy about using beta code anywhere), I'm only doing it this once because it's absolutely necessary to provide the functionality desired. -- Steve On Dec 11, 2008, at 2:53 PM, John Martyniak wrote: thanks for the advice. I just downloaded a completely clean version, haven't even tried to build it yet. Applied the same, and I received exactly the same results. Do you only apply the ivan patch 2? What version of patch are you running? -John On Dec 11, 2008, at 2:10 PM, Stephen Weiss wrote: Are you sure you have a clean copy of the source? Every time I've applied his patch I grab a fresh copy of the tarball and run the exact same command, it always works for me. Now, whether the collapsing actually works is a different matter... -- Steve On Dec 11, 2008, at 1:29 PM, John Martyniak wrote: Hi, I am trying to apply Ivan's field collapsing patch to solr 1.3 (not a nightly), and it continously fails. I am using the following command: patch -p0 -i collapsing-patch-to-1.3.0-ivan_2.patch --dry-run I am in the apache-solr directory, and have read write for all files directories and files. I am get the following results: patching file src/test/org/apache/solr/search/TestDocSet.java Hunk #1 FAILED at 88. 1 out of 1 hunk FAILED -- saving rejects to file src/test/org/ apache/solr/search/TestDocSet.java.rej patching file src/java/org/apache/solr/search/CollapseFilter.java patching file src/java/org/apache/solr/search/DocSet.java Hunk #1 FAILED at 195. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/ apache/solr/search/DocSet.java.rej patching file src/java/org/apache/solr/search/NegatedDocSet.java patching file src/java/org/apache/solr/search/ SolrIndexSearcher.java Hunk #1 FAILED at 1357. 1 out of 1 hunk FAILED -- saving rejects to file src/java/org/ apache/solr/search/SolrIndexSearcher.java.rej patching file src/java/org/apache/solr/common/params/ CollapseParams.java patching file src/java/org/apache/solr/handler/component/ CollapseComponent.java Also the '.rej' files are not created. Does anybody have any ideas? thanks in advance for the help. -John
RE: exceeded limit of maxWarmingSearchers=4
How big is your index? There is a variant of the Lucene disk accessors in the Lucene contrib area. It stores all of the index data directly in POJOs (Java objects) and does not marshal them into a disk-saveable format. The indexes are understandably larger, but all data added is automatically commited; there is no actual commit operation. -Original Message- From: Walter Underwood [mailto:wunderw...@netflix.com] Sent: Thursday, December 11, 2008 11:45 AM To: solr-user@lucene.apache.org Subject: Re: exceeded limit of maxWarmingSearchers=4 It sounds like you need real-time search, where documents are available in the next query. Solr doesn't do that. That is a pretty rare feature and must be designed in at the start. The usual workaround is to have a main index plus a small delta index and search both. Deletes have to be handled separately. The Sphinx docs have a description of how to do that with their engine: http://www.sphinxsearch.com/docs/current.html#live-updates wunder On 12/11/08 10:44 AM, "chip correra" wrote: > > > We¹re using Solr as a backend indexer/search engine to support > an AJAX based consumer application. Basically, when users of our > system create ³Documents² in our product, we commit right away, > because we want to immediately re-query and get counts back from Solr > to update the user¹s interface. Occasionally, usually under load, we see the following error... > > ERROR [IndexSubscription] error committing changes to solr > java.lang.RuntimeException: org.apache.solr.common.SolrException: > Error opening new searcher. exceeded limit of maxWarmingSearchers=4, > try again later. > > My understanding of Solr¹s caches and warming searches is that we get > nearly no value out of them because with many users, each only ever > querying their own data, the caches would quickly be marked irrelevant > by new unique user queries. > > So, we¹ve tried setting true > And we¹ve set all the cache autowarmCounts=²0², for example: > autowarmCount="0"/> > > Yet the same problem is periodically logged. > > Is there a way to disable warming Searches? > > Should we even be trying this? > > Is there a better configuration, i.e. Do we just set Warming Searches > to a higher number? > > I read in solr...@jira that removing this configuration all together > forces the number of warming Searches be infinite that can¹t be good? > > > _ > Send e-mail anywhere. No map, no compass. > http://windowslive.com/Explore/hotmail?ocid=TXT_TAGLM_WL_hotmail_acq_a > nywhere_ > 122008
importing csv into Solr
I can't import csv files into Solr - I've gone through the wiki and all the examples online, but I've hit the same error - what am I'm doing wrong? curl 'http://localhost:8080/solr/update/csv?commit=true' --data-binary @green.csv -H 'Content-type:text/plain; charset=utf-8' -s -u solrAdmin:solrAdmin Apache Tomcat/5.5.26 - Error report HTTP Status 400 - undefined field colortype Status reportmessage undefined field colordescription The request sent by the client was syntactically incorrect (undefined field color).Apache Tomcat/5.5.26[11:23:51] I've also tried to stream the csv in: curl http://localhost:8080/solr/update/csv?stream.file=green.csv -s -u solrAdmin:solrAdmin Apache Tomcat/5.5.26 - Error report HTTP Status 400 - undefined field colortype Status reportmessage undefined field colordescription The request sent by the client was syntactically incorrect (undefined field color).Apache Tomcat/5.5.26 Thanks P
Re: importing csv into Solr
The error message is saying "undefined field color" Is that field defined in your schema? If not, you need to define it, or map the color field to another field during import. -Yonik On Thu, Dec 11, 2008 at 11:37 PM, phil cryer wrote: > I can't import csv files into Solr - I've gone through the wiki and > all the examples online, but I've hit the same error - what am I'm > doing wrong? > > curl 'http://localhost:8080/solr/update/csv?commit=true' --data-binary > @green.csv -H 'Content-type:text/plain; charset=utf-8' -s -u > solrAdmin:solrAdmin > > Apache Tomcat/5.5.26 - Error > report HTTP Status 400 - undefined > field colortype Status > reportmessage undefined field > colordescription The request sent by the client > was syntactically incorrect (undefined field color). size="1" noshade="noshade">Apache > Tomcat/5.5.26[11:23:51] > > I've also tried to stream the csv in: > > curl http://localhost:8080/solr/update/csv?stream.file=green.csv -s -u > solrAdmin:solrAdmin > > Apache Tomcat/5.5.26 - Error > report HTTP Status 400 - undefined > field colortype Status > reportmessage undefined field > colordescription The request sent by the client > was syntactically incorrect (undefined field color). size="1" noshade="noshade">Apache Tomcat/5.5.26 > > Thanks > > P >
Re: SolrConfig.xml Replication
The moment the current patch is tested it will be checked in. On Thu, Dec 11, 2008 at 8:33 PM, Jeff Newburn wrote: > Thank you for the quick response. I will keep an eye on that to see how it > progresses. > > > On 12/10/08 8:03 PM, "Noble Paul നോബിള് नोब्ळ्" > wrote: > >> This is a known issue and I was planning to take it up soon. >> https://issues.apache.org/jira/browse/SOLR-821 >> >> >> On Thu, Dec 11, 2008 at 5:30 AM, Jeff Newburn wrote: >>> I am curious as to whether there is a solution to be able to replicate >>> solrconfig.xml with the 1.4 replication. The obvious problem is that the >>> master would replicate the solrconfig turning all slaves into masters with >>> its config. I have also tried on a whim to configure the master and slave >>> on the master so that the slave points to the same server but that seems to >>> break the replication completely. Please let me know if anybody has any >>> ideas >>> >>> -Jeff >>> >> >> > > -- --Noble Paul
Re: Error, when i update the rich text documents such as .doc, .ppt files.
I don't have time to verify this now, but the RichDocumentHandler does not have a separate contrib directory and I don't think the RichDocumentHandler patch makes a jar particular to the handler; instead, the java files get dumped in the main solr tree (java/org/apache/solr) , and therefore they get compiled into the main solr war when you do "ant example". I too would suggest looking at ExtractingRequestHandler rather than RichDocumentHandler, though. I could be wrong, but I don't think anyone necessarily has plans to maintain RichDocumentHandler going forward. Plus ExtractingRequestHandler has more features. On Wed, Dec 10, 2008 at 9:04 AM, Grant Ingersoll wrote: > Hi Raghav, > > Recently, integration with Tika was completed for SOLR-284 and it is now > committed on the trunk (but does not use the old RichDocumentHandler > approach). See http://wiki.apache.org/solr/ExtractingRequestHandler for how > to use and configure. > > Otherwise, it looks to me like the jar file for the RichDocHandler is not in > your WAR or in the Solr Home lib directory. > > HTH, > Grant > > On Dec 10, 2008, at 7:09 AM, RaghavPrabhu wrote: > >> >> Hi all, >> >> I want to index the rich text documents like .doc, .xls, .ppt files. I had >> done the patch for updating the rich documents by followed the >> instructions >> in this below url. http://wiki.apache.org/solr/UpdateRichDocuments >> >> When i indexing the doc file, im getting this following error in the >> browser.
