Re: Possible memory leaks with frequent replication

2010-11-03 Thread Lance Norskog
Do you use EmbeddedSolr in the query server? There is a memory leak that shows up when taking a lot of replications. On Wed, Nov 3, 2010 at 8:28 AM, Jonathan Rochkind wrote: > Ah, but reading Peter's email message I reference more carefully, it seems > that Solr already DOES provide an info-level

Re: Possible memory leaks with frequent replication

2010-11-03 Thread Jonathan Rochkind
Ah, but reading Peter's email message I reference more carefully, it seems that Solr already DOES provide an info-level log warning you about over-lapping warming, awesome. (But again, I'm pretty sure it does NOT throw or HTTP error in that condition, based on my and others experience). > To

Re: Possible memory leaks with frequent replication

2010-11-03 Thread Jonathan Rochkind
I hadn't looked at the code, am not familiar with Solr code, and can't say what that code does. But I have experienced issues that I _believe_ were caused by too frequent commits causing over-lapping searcher preperation. And I've definitely seen Solr documentation that suggests this is an iss

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Lance Norskog
Isn't that what this code does? onDeckSearchers++; if (onDeckSearchers < 1) { // should never happen... just a sanity check log.error(logid+"ERROR!!! onDeckSearchers is " + onDeckSearchers); onDeckSearchers=1; // reset } else if (onDeckSearchers > maxWarm

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Jonathan Rochkind
It's definitely a known 'issue' that you can't replicate (or do any other kind of index change, including a commit) at a faster frequency than your warming queries take to complete, or you'll wind up with something like you've seen. It's in some documentation somewhere I saw, for sure. The ad

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Yonik Seeley
On Tue, Nov 2, 2010 at 12:32 PM, Simon Wistow wrote: > On Mon, Nov 01, 2010 at 05:42:51PM -0700, Lance Norskog said: >> You should query against the indexer. I'm impressed that you got 5s >> replication to work reliably. > > That's our current solution - I was just wondering if there was anything

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Simon Wistow
On Mon, Nov 01, 2010 at 05:42:51PM -0700, Lance Norskog said: > You should query against the indexer. I'm impressed that you got 5s > replication to work reliably. That's our current solution - I was just wondering if there was anything I was missing. Thanks!

Re: Possible memory leaks with frequent replication

2010-11-01 Thread Lance Norskog
You should query against the indexer. I'm impressed that you got 5s replication to work reliably. On Mon, Nov 1, 2010 at 4:27 PM, Simon Wistow wrote: > We've been trying to get a setup in which a slave replicates from a > master every few seconds (ideally every second but currently we have it > s