Re: Possible memory leaks with frequent replication

2010-11-03 Thread Lance Norskog
Do you use EmbeddedSolr in the query server? There is a memory leak that shows up when taking a lot of replications. On Wed, Nov 3, 2010 at 8:28 AM, Jonathan Rochkind wrote: > Ah, but reading Peter's email message I reference more carefully, it seems > that Solr already DOES provide an info-level

Re: Possible memory leaks with frequent replication

2010-11-03 Thread Jonathan Rochkind
Ah, but reading Peter's email message I reference more carefully, it seems that Solr already DOES provide an info-level log warning you about over-lapping warming, awesome. (But again, I'm pretty sure it does NOT throw or HTTP error in that condition, based on my and others experience). > To

Re: Possible memory leaks with frequent replication

2010-11-03 Thread Jonathan Rochkind
I hadn't looked at the code, am not familiar with Solr code, and can't say what that code does. But I have experienced issues that I _believe_ were caused by too frequent commits causing over-lapping searcher preperation. And I've definitely seen Solr documentation that suggests this is an iss

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Lance Norskog
Isn't that what this code does? onDeckSearchers++; if (onDeckSearchers < 1) { // should never happen... just a sanity check log.error(logid+"ERROR!!! onDeckSearchers is " + onDeckSearchers); onDeckSearchers=1; // reset } else if (onDeckSearchers > maxWarm

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Jonathan Rochkind
It's definitely a known 'issue' that you can't replicate (or do any other kind of index change, including a commit) at a faster frequency than your warming queries take to complete, or you'll wind up with something like you've seen. It's in some documentation somewhere I saw, for sure. The ad

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Yonik Seeley
On Tue, Nov 2, 2010 at 12:32 PM, Simon Wistow wrote: > On Mon, Nov 01, 2010 at 05:42:51PM -0700, Lance Norskog said: >> You should query against the indexer. I'm impressed that you got 5s >> replication to work reliably. > > That's our current solution - I was just wondering if there was anything

Re: Possible memory leaks with frequent replication

2010-11-02 Thread Simon Wistow
On Mon, Nov 01, 2010 at 05:42:51PM -0700, Lance Norskog said: > You should query against the indexer. I'm impressed that you got 5s > replication to work reliably. That's our current solution - I was just wondering if there was anything I was missing. Thanks!

Re: Possible memory leaks with frequent replication

2010-11-01 Thread Lance Norskog
You should query against the indexer. I'm impressed that you got 5s replication to work reliably. On Mon, Nov 1, 2010 at 4:27 PM, Simon Wistow wrote: > We've been trying to get a setup in which a slave replicates from a > master every few seconds (ideally every second but currently we have it > s

Possible memory leaks with frequent replication

2010-11-01 Thread Simon Wistow
We've been trying to get a setup in which a slave replicates from a master every few seconds (ideally every second but currently we have it set at every 5s). Everything seems to work fine until, periodically, the slave just stops responding from what looks like it running out of memory: org.ap