Re: Throughput Optimization

2008-12-04 Thread Yonik Seeley
On Thu, Dec 4, 2008 at 2:30 PM, wojtekpia <[EMAIL PROTECTED]> wrote: > > New faceting stuff off because I'm encountering some problems when I turn it > on, I posted the details: > http://www.nabble.com/new-faceting-algorithm-td20674902.html#a20840622 Missed that, thanks... will respond there. -Yo

Re: Throughput Optimization

2008-12-04 Thread wojtekpia
New faceting stuff off because I'm encountering some problems when I turn it on, I posted the details: http://www.nabble.com/new-faceting-algorithm-td20674902.html#a20840622 Yonik Seeley wrote: > > On Thu, Dec 4, 2008 at 1:54 PM, wojtekpia <[EMAIL PROTECTED]> wrote: >> It looks like file lockin

Re: Throughput Optimization

2008-12-04 Thread Yonik Seeley
On Thu, Dec 4, 2008 at 1:54 PM, wojtekpia <[EMAIL PROTECTED]> wrote: > It looks like file locking was the bottleneck - CPU usage is up to ~98% (from > the previous peak of ~50%). Great to hear it! > I'm running the trunk code from Dec 2 with the > faceting improvement (SOLR-475) turned off. Thank

Re: Throughput Optimization

2008-12-04 Thread wojtekpia
It looks like file locking was the bottleneck - CPU usage is up to ~98% (from the previous peak of ~50%). I'm running the trunk code from Dec 2 with the faceting improvement (SOLR-475) turned off. Thanks for all the help! Yonik Seeley wrote: > > FYI, SOLR-465 has been committed. Let us know if

Re: Throughput Optimization

2008-11-07 Thread Yonik Seeley
FYI, SOLR-465 has been committed. Let us know if it improves your scenario. -Yonik On Wed, Nov 5, 2008 at 5:39 PM, Yonik Seeley <[EMAIL PROTECTED]> wrote: > On Wed, Nov 5, 2008 at 5:18 PM, wojtekpia <[EMAIL PROTECTED]> wrote: >> I'd like to integrate this improvement into my deployment. Is it ju

Re: Throughput Optimization

2008-11-05 Thread Yonik Seeley
On Wed, Nov 5, 2008 at 5:18 PM, wojtekpia <[EMAIL PROTECTED]> wrote: > I'd like to integrate this improvement into my deployment. Is it just a > matter of getting the latest Lucene jars (Lucene nightly build)? You need to apply this source code patch to Solr: https://issues.apache.org/jira/browse/

Re: Throughput Optimization

2008-11-05 Thread wojtekpia
I'd like to integrate this improvement into my deployment. Is it just a matter of getting the latest Lucene jars (Lucene nightly build)? Yonik Seeley wrote: > > You're probably hitting some contention with the locking around the > reading of index files... this has been recently improved in Luc

Re: Throughput Optimization

2008-11-05 Thread Yonik Seeley
On Wed, Nov 5, 2008 at 2:44 PM, wojtekpia <[EMAIL PROTECTED]> wrote: > I'll try changing my other caches to LRUCache and observe performance. > Interestingly, the FastLRUCache has given me a ~10% increase in performance, > much lower than I've read on the SOLR-667 thread. That's better than I woul

Re: Throughput Optimization

2008-11-05 Thread Yonik Seeley
On Wed, Nov 5, 2008 at 2:29 PM, Feak, Todd <[EMAIL PROTECTED]> wrote: > Yonik said something about the FastLRUCache giving the most gain for > high hit-rates and the LRUCache being faster for low hit-rates. Right, for single-threaded requests. FastLRUCache has faster gets and slower puts (on aver

RE: Throughput Optimization

2008-11-05 Thread wojtekpia
I'll try changing my other caches to LRUCache and observe performance. Interestingly, the FastLRUCache has given me a ~10% increase in performance, much lower than I've read on the SOLR-667 thread. Would compressing some of my stored fields significantly improve performance? Most of my stored fie

RE: Throughput Optimization

2008-11-05 Thread Feak, Todd
curious. -Todd Feak -Original Message- From: wojtekpia [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 05, 2008 11:08 AM To: solr-user@lucene.apache.org Subject: RE: Throughput Optimization My documentCache hit rate is ~.7, and my queryCache is ~.03. I'm using FastLRUCache on al

RE: Throughput Optimization

2008-11-05 Thread wojtekpia
iginal Message- > From: wojtekpia [mailto:[EMAIL PROTECTED] > Sent: Wednesday, November 05, 2008 8:15 AM > To: solr-user@lucene.apache.org > Subject: Re: Throughput Optimization > > > Yes, I am seeing evictions. I've tried setting my filterCache higher, > but > then I

Re: Throughput Optimization

2008-11-05 Thread wojtekpia
Where is the alt directory in the source tree (or what is the JIRA issue number)? I'd like to apply this patch and re-run my tests. Does changing the lockType in solrconfig.xml address this issue? (My lockType is the default - single). markrmiller wrote: > > The latest alt directory patch uses

RE: Throughput Optimization

2008-11-05 Thread Feak, Todd
What are your other cache hit rates looking like? Which caches are you using the FastLRUCache on? -Todd Feak -Original Message- From: wojtekpia [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 05, 2008 8:15 AM To: solr-user@lucene.apache.org Subject: Re: Throughput Optimization Yes

Re: Throughput Optimization

2008-11-05 Thread Yonik Seeley
On Wed, Nov 5, 2008 at 11:14 AM, wojtekpia <[EMAIL PROTECTED]> wrote: > Yes, I am seeing evictions. I've tried setting my filterCache higher, but > then I start getting Out Of Memory exceptions. My filterCache hit ratio is > > .99. It looks like I've hit a RAM bound here. Evictions on the filterCa

RE: Throughput Optimization

2008-11-05 Thread Feak, Todd
o:[EMAIL PROTECTED] Sent: Wednesday, November 05, 2008 8:15 AM To: solr-user@lucene.apache.org Subject: Re: Throughput Optimization Yes, I am seeing evictions. I've tried setting my filterCache higher, but then I start getting Out Of Memory exceptions. My filterCache hit ratio is > .99. It

Re: Throughput Optimization

2008-11-05 Thread christophe
Does the number of searcher affect CPU usage ? Not totally sure about it but I think some versions of Tomcat were not totally scalable over 4 CPUs (or 4 cores). C. wojtekpia wrote: Yes, I am seeing evictions. I've tried setting my filterCache higher, but then I start getting Out Of Memory exc

Re: Throughput Optimization

2008-11-05 Thread wojtekpia
Yes, I am seeing evictions. I've tried setting my filterCache higher, but then I start getting Out Of Memory exceptions. My filterCache hit ratio is > .99. It looks like I've hit a RAM bound here. I ran a test without faceting. The response times / throughput were both significantly higher, there

Re: Throughput Optimization

2008-11-05 Thread Mark Miller
The latest alt directory patch uses It. - Mark On Nov 5, 2008, at 9:25 AM, "Yonik Seeley" <[EMAIL PROTECTED]> wrote: You're probably hitting some contention with the locking around the reading of index files... this has been recently improved in Lucene for non-Windows boxes, and we're integra

Re: Throughput Optimization

2008-11-05 Thread Yonik Seeley
You're probably hitting some contention with the locking around the reading of index files... this has been recently improved in Lucene for non-Windows boxes, and we're integrating that into Solr (should def be in the next release). -Yonik On Tue, Nov 4, 2008 at 9:01 PM, wojtekpia <[EMAIL PROTECT

Re: Throughput Optimization

2008-11-05 Thread Erik Hatcher
One quick question are you seeing any evictions from your filterCache? If so, it isn't set large enough to handle the faceting you're doing. Erik On Nov 4, 2008, at 8:01 PM, wojtekpia wrote: I've been running load tests over the past week or 2, and I can't figure out my sy