Errors compiling laster solr 1.3 update
Hi all, First of all , sorry for my English I'm not sure that it's a problem, but after the last update from CVS (solr 1.3 dev) I can't compile the solr source. I think that is a problema of my workspace, but I'd like to be sure that anyone more have the same problema. The classes who have the problema are SnowballPorterFilterFactory and SolrCore Thanks Raul
RE: Errors compiling laster solr 1.3 update
Hi Shalin, I too think that is a problem of jars files , but I download the lib directory again and isn't work. This is my CVS link http://svn.apache.org/repos/asf/lucene/solr/trunk and y too try whith http://svn.apache.org/repos/asf/lucene/solr/branches/branch-1.3/ It`s correct ??? -Mensaje original- De: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED] Enviado el: miércoles, 03 de septiembre de 2008 18:56 Para: solr-user@lucene.apache.org Asunto: Re: Errors compiling laster solr 1.3 update I can compile it successfully. The lucene jars have been updated, so make sure you update the lib directory too. On Wed, Sep 3, 2008 at 9:30 PM, <[EMAIL PROTECTED]> wrote: > Hi all, > > > > First of all , sorry for my English > > > > I'm not sure that it's a problem, but after the last update from CVS (solr > 1.3 dev) I can't compile the solr source. I think that is a problema of my > workspace, but I'd like to be sure that anyone more have the same problema. > > The classes who have the problema are SnowballPorterFilterFactory and > SolrCore > > Thanks > > > > Raul > > -- Regards, Shalin Shekhar Mangar.
RE: Errors compiling laster solr 1.3 update
OK that's the problema :-) I forget to update the WebContent Libs Thanks all -Mensaje original- De: Grant Ingersoll [mailto:[EMAIL PROTECTED] Enviado el: miércoles, 03 de septiembre de 2008 21:04 Para: solr-user@lucene.apache.org Asunto: Re: Errors compiling laster solr 1.3 update Did you run clean first? Can you share the errors? Note, it compiles for me. On Sep 3, 2008, at 2:15 PM, <[EMAIL PROTECTED]> <[EMAIL PROTECTED]> wrote: > Hi Shalin, > I too think that is a problem of jars files , but I download the lib > directory again and isn't work. > This is my CVS link http://svn.apache.org/repos/asf/lucene/solr/ > trunk and y > too try whith > http://svn.apache.org/repos/asf/lucene/solr/branches/branch-1.3/ > > > It`s correct ??? > > -Mensaje original- > De: Shalin Shekhar Mangar [mailto:[EMAIL PROTECTED] > Enviado el: miércoles, 03 de septiembre de 2008 18:56 > Para: solr-user@lucene.apache.org > Asunto: Re: Errors compiling laster solr 1.3 update > > I can compile it successfully. The lucene jars have been updated, so > make > sure you update the lib directory too. > > On Wed, Sep 3, 2008 at 9:30 PM, <[EMAIL PROTECTED]> wrote: > >> Hi all, >> >> >> >> First of all , sorry for my English >> >> >> >> I'm not sure that it's a problem, but after the last update from >> CVS (solr >> 1.3 dev) I can't compile the solr source. I think that is a >> problema of my >> workspace, but I'd like to be sure that anyone more have the same > problema. >> >> The classes who have the problema are SnowballPorterFilterFactory and >> SolrCore >> >> Thanks >> >> >> >> Raul >> >> > > > -- > Regards, > Shalin Shekhar Mangar. >
RE: What's the bottleneck?
Hi Jason, some questions .. what is your index configuration??? What is your average size form the returned fields ??? How much memory have your System ?? Do you have long fieds who is returned in the queries ? Do you have actívate the Highlighting in the request ? Are you using multi-value filed for filter ... -Mensaje original- De: Jason Rennie [mailto:[EMAIL PROTECTED] Enviado el: jueves, 11 de septiembre de 2008 17:25 Para: solr-user@lucene.apache.org Asunto: What's the bottleneck? We have a 14 million document index that we only use for querying (optimized, read-only). When we issue queries that have few, relatively rare words, the query returns quickly. However, when the query is longer and uses more common words (hitting, say, ~1 million docs), it might take seconds to return. I'd like to know: what's the bottleneck? It doesn't seem to be disk---i/o wait times on the machine are much, much lower than on our database servers (e.g. 3% vs. 50%). Our search server is an 8-core machine and we do see cpu regularly holding above 100%, so cpu seems plausible, but would it really take that long to compute scores? We're using DisMax. There are a number of different fields that we search over (5 to be exact). We also have an fq on a single-digit status field. Does it make sense that computation time could easily exceed a second? If cpu is the bottleneck, is there anything we could do to easily trim-down computation time (besides removing common words from the query)? Jason -- Jason Rennie Head of Machine Learning Technologies, StyleFeeder http://www.stylefeeder.com/ Samantha's blog & pictures: http://samanthalyrarennie.blogspot.com/
RE: What's the bottleneck?
Ok, have you a average size about the memory ocupation, by Solr ? You must to have a look about the really memory usage from cached fields, and try to set java memory to upper value Are you evaluate the performance factors: http://wiki.apache.org/solr/SolrPerformanceFactors I think that is a memory problem, because when you issue queries that have few documents, its are load into memory (cache from solr) and the next queries don't use IO disk operation. But when the queries return too many documents, its can't be load in memory, by size, and for any query the solr must do load/unload memory operations, and disk reads ... Other cause can be the Lucene memory ocupation, but i need know what is the realy memory ocupation for the index. Sorry for my english :-( -Mensaje original- De: Jason Rennie [mailto:[EMAIL PROTECTED] Enviado el: jueves, 11 de septiembre de 2008 21:58 Para: solr-user@lucene.apache.org Asunto: Re: What's the bottleneck? On Thu, Sep 11, 2008 at 1:29 PM, <[EMAIL PROTECTED]> wrote: > what is your index configuration??? Not sure what you mean. We're using 1.2, though we've tested with a recent nightly and didn't see a significant change in performance... > What is your average size form the returned fields ??? Returned fields are relatively small, ~200 characters total per document. We're requesting the top 10 or so docs. How much memory have your System ?? 8g. We give the jvm a 2g (max) heap. We have another solr running on the same box also w/ 2g heap. The Linux kernel caches ~2.5g of disk. > Do you have long fieds who is returned in the queries ? No. The searched and returned fields are relatively short. One searched-over (but not returned) field can get up to a few hundred characters, but it's safe to assume they're all < 1k. > Do you have actívate the Highlighting in the request ? Nope. > Are you using multi-value filed for filter ... No, it does not have the multiValue attribute turned on. The qf field is just an integer. Any thoughts/comments are appreciated. Thanks, Jason
RE: What's the bottleneck?
Hi Jason, I'd like to know how you solved the problem. could you post the solution?? Thanks Raúl -Mensaje original- De: Jason Rennie [mailto:[EMAIL PROTECTED] Enviado el: jueves, 11 de septiembre de 2008 21:58 Para: solr-user@lucene.apache.org Asunto: Re: What's the bottleneck? On Thu, Sep 11, 2008 at 1:29 PM, <[EMAIL PROTECTED]> wrote: > what is your index configuration??? Not sure what you mean. We're using 1.2, though we've tested with a recent nightly and didn't see a significant change in performance... > What is your average size form the returned fields ??? Returned fields are relatively small, ~200 characters total per document. We're requesting the top 10 or so docs. How much memory have your System ?? 8g. We give the jvm a 2g (max) heap. We have another solr running on the same box also w/ 2g heap. The Linux kernel caches ~2.5g of disk. > Do you have long fieds who is returned in the queries ? No. The searched and returned fields are relatively short. One searched-over (but not returned) field can get up to a few hundred characters, but it's safe to assume they're all < 1k. > Do you have actívate the Highlighting in the request ? Nope. > Are you using multi-value filed for filter ... No, it does not have the multiValue attribute turned on. The qf field is just an integer. Any thoughts/comments are appreciated. Thanks, Jason
RE: Out of Memory Errors
Hi Willie, Are you using highliting ??? If, the response is yes, you need to know that for each document retrieved, the solr highliting load into memory the full field who is using for this functionality. If the field is too long, you have problems with memory. You can solve the problem using this patch: http://mail-archives.apache.org/mod_mbox/lucene-solr-dev/200806.mbox/%3C1552 [EMAIL PROTECTED] to copy the content of the field who is used to highliting to another field and reduce the size. You also need to know too that Windows have a limitation for memory process in 2 GB. -Mensaje original- De: Willie Wong [mailto:[EMAIL PROTECTED] Enviado el: miércoles, 22 de octubre de 2008 3:48 Para: solr-user@lucene.apache.org Asunto: Out of Memory Errors Hello, I've been having issues with out of memory errors on searches in Solr. I was wondering if I'm hitting a limit with solr or if I've configured something seriously wrong. Solr Setup - 3 cores - 3163615 documents each - 10 GB size - approx 10 fields - document sizes vary from a few kb to a few MB - no faceting is used however the search query can be fairly complex with 8 or more fields being searched on at once Environment: - windows 2003 - 2.8 GHz zeon processor - 1.5 GB memory assigned to solr - Jetty 6 server Once we get to around a few concurrent users OOM start occuring and Jetty restarts. Would this just be a case of more memory or are there certain configuration settings that need to be set? We're using an out of the box Solr 1.3 beta version. A few of the things we considered that might help: - Removing sorts on the result sets (result sets are approx 40,000 + documents) - Reducing cache sizes such as the queryResultMaxDocsCached setting, document cache, queryResultCache, filterCache, etc Am I missing anything else that should be looked at, or is it time to simply increase the memory/start looking at distributing the indexes? Any help would be much appreciated. Regards, WW
