Aleksey,
It was a less than ideal situation. because we did not have a choice. We
had external systems/scripts to manage this. A new custom implementation is
being built on SolrCloud which would have taken care of most of hose
issues.
SolrReplication is a hidden once you move to cloud. But it wi
bject: Re: LotsOfCores feature
On Fri, Jun 7, 2013, at 02:59 PM, Jack Krupansky wrote:
AFAICT, SolrCloud addresses the use case of distributed update for a
relatively smaller number of collections (dozens?) that have a relatively
larger number of rows - billions over a modest to moderate number of
On Fri, Jun 7, 2013, at 02:59 PM, Jack Krupansky wrote:
> AFAICT, SolrCloud addresses the use case of distributed update for a
> relatively smaller number of collections (dozens?) that have a relatively
> larger number of rows - billions over a modest to moderate number of
> nodes
> (a handful
Thanks Paul. Just a little clarification:
You mention that you migrate data using built-in replication, but if
you map and route users yourself, doesn't that mean that you also need
to manage replication yourself? Your routing logic needs to be aware
of how to map both replicas for each user, and
We set it up like this
+ individual solr instances are setup
+ external mapping/routing to allocate users to instances. This information
can be stored in an external data store
+ all cores are created as transient and loadonstart as false
+ cores come online on demand
+ as and when users data get b
fair number are online for short
periods of time.
-- Jack Krupansky
-Original Message-
From: Aleksey
Sent: Friday, June 07, 2013 3:44 PM
To: solr-user
Subject: Re: LotsOfCores feature
Aleksey: What would you say is the average core size for your use case -
thousands or millions of rows?
> Aleksey: What would you say is the average core size for your use case -
> thousands or millions of rows? And how sharded would each of your
> collections be, if at all?
Average core/collection size wouldn't even be thousands, hundreds more
like. And the largest would be half a million or so but
:38 AM
To: solr-user@lucene.apache.org
Subject: Re: LotsOfCores feature
The Wiki page was built not for Cloud Solr.
We have done such a deployment where less than a tenth of cores were active
at any given point in time. though there were tens of million indices they
were split among a large no:of
I should have been clearer, and others have mentioned... the "lots of cores"
stuff is really outside Zookeeper/SolrCloud at present. I don't think it's
incompatible, but it wasn't part of the design so it'll need some effort to
make it play nice with SolrCloud. I'm not sure there's actually a compe
The Wiki page was built not for Cloud Solr.
We have done such a deployment where less than a tenth of cores were active
at any given point in time. though there were tens of million indices they
were split among a large no:of hosts.
If you don't insist of Cloud deployment it is possible. I'm not
> A use case would a web site or service that had millions of users, each of
> whom would have an active Solr core when they are active, but inactive
> otherwise. Of course those cores would not all reside on one node and
> ZooKeeper is out of the question for managing anything that is in the
> mil
On 6/6/2013 6:32 PM, Jack Krupansky wrote:
> This would be a lot more of a true "Solr Cloud" than the "cluster"
> support that we have today.
>
> And the "CloudKeeper" itself might be a "traditional" SolrCloud cluster,
> except that it needs to be multi-data center.
I like a lot of what you sa
and notify CK that it is inactive.
Or something like that.
This would be a lot more of a true "Solr Cloud" than the "cluster" support
that we have today.
And the "CloudKeeper" itself might be a "traditional" SolrCloud cluster,
except that it needs
M, Jack Krupansky
> wrote:
>> So, is that a clear yes or a clear no for Aleksey's use case - 10's of
>> millions of cores, not all active but each loadable on demand?
>>
>> I asked this same basic question months ago and there was no answer
>> forthcoming.
>>
&
asic question months ago and there was no answer
> forthcoming.
>
> -- Jack Krupansky
>
> -Original Message- From: Erick Erickson
> Sent: Thursday, June 06, 2013 3:53 PM
> To: solr-user@lucene.apache.org
> Subject: Re: LotsOfCores feature
>
>
> 100K is rea
kson
Sent: Thursday, June 06, 2013 3:53 PM
To: solr-user@lucene.apache.org
Subject: Re: LotsOfCores feature
100K is really not the limit, it's just hard to imagine
100K cores on a single machine unless some were
really rarely used. And it's per node, not cluster-wide.
The current state is t
100K is really not the limit, it's just hard to imagine
100K cores on a single machine unless some were
really rarely used. And it's per node, not cluster-wide.
The current state is that everything is in place, including
transient cores, auto-discovery, etc. So you should be
able to go ahead and t
17 matches
Mail list logo
