core on the fly, based on the existence of the core on the disk
> (absolute path calculated from the core name).
> >
> > Thanks for your interest,
> >
> > Olivier
> >
> > De : Erick Erickson
> > [erickerick...@gmail.co
core entries in
> solr.xml, and we will use the cores Auto option to create load or only load
> the core on the fly, based on the existence of the core on the disk
> (absolute path calculated from the core name).
> >
> > Thanks for your interest,
> >
> > Olivier
>
s Auto option to create load or only load the core on
> the fly, based on the existence of the core on the disk (absolute path
> calculated from the core name).
>
> Thanks for your interest,
>
> Olivier
>
> De : Erick Erickson [erickerick...@gmail.com<mailto:erickeric
the core name).
>
> Thanks for your interest,
>
> Olivier
> ________
> De : Erick Erickson [erickerick...@gmail.com]
> Date d'envoi : lundi 7 octobre 2013 14:33
> À : solr-user@lucene.apache.org
> Objet : Re: feedback on Solr 4.x Lots
mail.com]
Date d'envoi : lundi 7 octobre 2013 14:33
À : solr-user@lucene.apache.org
Objet : Re: feedback on Solr 4.x LotsOfCores feature
Thanks for the great writeup! It's always interesting to see how
a feature plays out "in the real world". A couple of questions
though:
bq: W
> better scoring, some requests that do not support filtering, we have
> decided to use the LotsOfCores feature.
>
> Our goal was to change the current I/O usage : from lots of random I/O
> access on huge segments to mostly sequential I/O access on small segments.
> For our use case, it'
.fdt files)
> > Retrieving these fields from the index was costly, because of many seek
> > in large files, and no limit usage possible.
> > There is also an overhead on queries : too many results are filtered to
> > find only results concerning user.
> > For these re
onse time
> (Mean), x5.4 faster (95th per)
> - Delete 5 documents (avg) : x8.4 faster response time (Mean) x7.4
> faster (95th per)
> - Search : x3.7 faster response time (Mean) 4x faster (95th per)
>
> In fact, the better performance is mainly due to the little size of
decided to use the LotsOfCores feature.
Our goal was to change the current I/O usage : from lots of random I/O
access on huge segments to mostly sequential I/O access on small segments.
For our use case, it's not a big deal, that the first query to one not
yet loaded core will be slow.
And, we
Aleksey,
It was a less than ideal situation. because we did not have a choice. We
had external systems/scripts to manage this. A new custom implementation is
being built on SolrCloud which would have taken care of most of hose
issues.
SolrReplication is a hidden once you move to cloud. But it wi
bject: Re: LotsOfCores feature
On Fri, Jun 7, 2013, at 02:59 PM, Jack Krupansky wrote:
AFAICT, SolrCloud addresses the use case of distributed update for a
relatively smaller number of collections (dozens?) that have a relatively
larger number of rows - billions over a modest to moderate number of
On Fri, Jun 7, 2013, at 02:59 PM, Jack Krupansky wrote:
> AFAICT, SolrCloud addresses the use case of distributed update for a
> relatively smaller number of collections (dozens?) that have a relatively
> larger number of rows - billions over a modest to moderate number of
> nodes
> (a handful
Thanks Paul. Just a little clarification:
You mention that you migrate data using built-in replication, but if
you map and route users yourself, doesn't that mean that you also need
to manage replication yourself? Your routing logic needs to be aware
of how to map both replicas for each user, and
We set it up like this
+ individual solr instances are setup
+ external mapping/routing to allocate users to instances. This information
can be stored in an external data store
+ all cores are created as transient and loadonstart as false
+ cores come online on demand
+ as and when users data get b
fair number are online for short
periods of time.
-- Jack Krupansky
-Original Message-
From: Aleksey
Sent: Friday, June 07, 2013 3:44 PM
To: solr-user
Subject: Re: LotsOfCores feature
Aleksey: What would you say is the average core size for your use case -
thousands or millions of rows?
> Aleksey: What would you say is the average core size for your use case -
> thousands or millions of rows? And how sharded would each of your
> collections be, if at all?
Average core/collection size wouldn't even be thousands, hundreds more
like. And the largest would be half a million or so but
:38 AM
To: solr-user@lucene.apache.org
Subject: Re: LotsOfCores feature
The Wiki page was built not for Cloud Solr.
We have done such a deployment where less than a tenth of cores were active
at any given point in time. though there were tens of million indices they
were split among a large no:of
I should have been clearer, and others have mentioned... the "lots of cores"
stuff is really outside Zookeeper/SolrCloud at present. I don't think it's
incompatible, but it wasn't part of the design so it'll need some effort to
make it play nice with SolrCloud. I'm not sure there's actually a compe
The Wiki page was built not for Cloud Solr.
We have done such a deployment where less than a tenth of cores were active
at any given point in time. though there were tens of million indices they
were split among a large no:of hosts.
If you don't insist of Cloud deployment it is possible. I'm not
> A use case would a web site or service that had millions of users, each of
> whom would have an active Solr core when they are active, but inactive
> otherwise. Of course those cores would not all reside on one node and
> ZooKeeper is out of the question for managing anything that is in the
> mil
On 6/6/2013 6:32 PM, Jack Krupansky wrote:
> This would be a lot more of a true "Solr Cloud" than the "cluster"
> support that we have today.
>
> And the "CloudKeeper" itself might be a "traditional" SolrCloud cluster,
> except that it needs to be multi-data center.
I like a lot of what you sa
and notify CK that it is inactive.
Or something like that.
This would be a lot more of a true "Solr Cloud" than the "cluster" support
that we have today.
And the "CloudKeeper" itself might be a "traditional" SolrCloud cluster,
except that it needs
M, Jack Krupansky
> wrote:
>> So, is that a clear yes or a clear no for Aleksey's use case - 10's of
>> millions of cores, not all active but each loadable on demand?
>>
>> I asked this same basic question months ago and there was no answer
>> forthcoming.
>>
&
asic question months ago and there was no answer
> forthcoming.
>
> -- Jack Krupansky
>
> -Original Message- From: Erick Erickson
> Sent: Thursday, June 06, 2013 3:53 PM
> To: solr-user@lucene.apache.org
> Subject: Re: LotsOfCores feature
>
>
> 100K is rea
kson
Sent: Thursday, June 06, 2013 3:53 PM
To: solr-user@lucene.apache.org
Subject: Re: LotsOfCores feature
100K is really not the limit, it's just hard to imagine
100K cores on a single machine unless some were
really rarely used. And it's per node, not cluster-wide.
The current state is t
100K is really not the limit, it's just hard to imagine
100K cores on a single machine unless some were
really rarely used. And it's per node, not cluster-wide.
The current state is that everything is in place, including
transient cores, auto-discovery, etc. So you should be
able to go ahead and t
I was looking at this wiki and linked issues:
http://wiki.apache.org/solr/LotsOfCores
they talk about a limit being 100K cores. Is that per server or per
entire fleet because zookeeper needs to manage that?
I was considering a use case where I have tens of millions of indices
but less that a mill
decided.
Thanks & Regards,
Umesh
--
View this message in context:
http://lucene.472066.n3.nabble.com/Is-anobdy-using-lotsofcores-feature-in-production-tp3193798p3236958.html
Sent from the Solr - User mailing list archive at Nabble.com.
e around 1000 core and want to use this feature. Will
> > >
> > > there
> > >
> > > > be any issue in production?
> > > >
> > > > http://wiki.apache.org/solr/LotsOfCores
> > > >
> > > &g
re
> >
> > > be any issue in production?
> > >
> > > http://wiki.apache.org/solr/LotsOfCores
> > >
> > > Thanks,
> > > Umesh
> > >
> > > --
> >
> > > View this message in context:
> > http://lucene.472066.n3.nabble.com/Is-anobdy-using-lotsofcores-feature-in
> > -
> >
> > > production-tp3193798p3193798.html Sent from the Solr - User mailing
> > > list archive at Nabble.com.
roduction? Is this feature
> > scalable. I have around 1000 core and want to use this feature. Will
> there
> > be any issue in production?
> >
> > http://wiki.apache.org/solr/LotsOfCores
> >
> > Thanks,
> > Umesh
> >
> > --
> > View
t; --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Is-anobdy-using-lotsofcores-feature-in-
> production-tp3193798p3193798.html Sent from the Solr - User mailing list
> archive at Nabble.com.
.nabble.com/Is-anobdy-using-lotsofcores-feature-in-production-tp3193798p3193798.html
Sent from the Solr - User mailing list archive at Nabble.com.
33 matches
Mail list logo
