AW: AW: AW: Scaling to large Number of Collections

2014-09-01 Thread Christoph Schmidt
Montag, 1. September 2014 14:50 An: solr-user@lucene.apache.org Betreff: Re: AW: Scaling to large Number of Collections And I would add another suggested requirement - "dormant collections" - collections which may once have been active, but have not seen any recent activity and can h

Re: AW: Scaling to large Number of Collections

2014-09-01 Thread Jack Krupansky
r? -- Jack Krupansky -Original Message- From: Christoph Schmidt Sent: Monday, September 1, 2014 3:50 AM To: solr-user@lucene.apache.org Subject: AW: Scaling to large Number of Collections Yes, this would help us in our scenario. -Ursprüngliche Nachricht- Von: Jack Krupansky [mailto

AW: Scaling to large Number of Collections

2014-09-01 Thread Christoph Schmidt
We already reduced the -Xss256k. How could we reduce the size of the transaction log? By less autoCommits? Or could it be cleaned up? Thanks Christoph -Ursprüngliche Nachricht- Von: Shalin Shekhar Mangar [mailto:shalinman...@gmail.com] Gesendet: Sonntag, 31. August 2014 20:12 An: solr

AW: Scaling to large Number of Collections

2014-09-01 Thread Christoph Schmidt
Yes, this would help us in our scenario. -Ursprüngliche Nachricht- Von: Jack Krupansky [mailto:j...@basetechnology.com] Gesendet: Sonntag, 31. August 2014 18:10 An: solr-user@lucene.apache.org Betreff: Re: Scaling to large Number of Collections We should also consider "lightly-sharded" c

AW: Scaling to large Number of Collections

2014-09-01 Thread Christoph Schmidt
At peak time a lot of customers will be on the system. Sure not all in the same second but within an hour we see a lot of them. So having something like "stand-by" cores will help to reduce resource consumption (memory, threads) if the core are rarely used. Doing a warmup at login time of the u

AW: Scaling to large Number of Collections

2014-09-01 Thread Christoph Schmidt
Hm, temporarily more threads is hard. We already reduced -Xss256k. Wouldn't it be better to use Callable and Executor as proposed in: http://stackoverflow.com/questions/16789288/java-lang-outofmemoryerror-unable-to-create-new-native-thread and limit the number of used threads to the number of CP

AW: Scaling to large Number of Collections

2014-09-01 Thread Christoph Schmidt
Yes, we will think about this how to reorganise the application. Thanks Christoph -Ursprüngliche Nachricht- Von: Joseph Obernberger [mailto:joseph.obernber...@gmail.com] Gesendet: Sonntag, 31. August 2014 16:58 An: solr-user@lucene.apache.org Betreff: Re: Scaling to large Number of Colle

AW: Scaling to large Number of Collections

2014-09-01 Thread Christoph Schmidt
Is there a Jira task for this? Thanks Christoph -Ursprüngliche Nachricht- Von: Mark Miller [mailto:markrmil...@gmail.com] Gesendet: Sonntag, 31. August 2014 14:24 An: solr-user Betreff: Re: Scaling to large Number of Collections > On Aug 31, 2014, at 4:04 AM, Christoph Schmidt > wrot

Re: AW: Scaling to large Number of Collections

2014-08-31 Thread Jack Krupansky
From: Christoph Schmidt Sent: Sunday, August 31, 2014 11:44 AM To: solr-user@lucene.apache.org Subject: AW: Scaling to large Number of Collections One collection has 2 replicas, no sharding, the collections are not that big. No, they are unfortunately not independent. There are collec

AW: Scaling to large Number of Collections

2014-08-31 Thread Christoph Schmidt
One collection has 2 replicas, no sharding, the collections are not that big. No, they are unfortunately not independent. There are collections with customer documents (some thousand customers) and product collections. One customer has at least on customer collection and 1 to some hundred produc