uction hardware now,
> before profiling - I get to control much of the design and construction so
> I don't argue with this!)
>
> Thanks for all the comments everyone, all very much appreciated :)
> Gil
>
>
> -Original Message-
> From: Toke Eskildsen [mailto:t..
On Thu, 2013-12-12 at 11:10 +0100, Hoggarth, Gil wrote:
> Thanks for this - I haven't any previous experience with utilising SSDs
> in the way you suggest, so I guess I need to start learning!
There's a bit of divide in the Lucene/Solr-world on this. Everybody
agrees that SSDs in themselves are gr
profiling - I get to
control much of the design and construction so I don't argue with this!)
Thanks for all the comments everyone, all very much appreciated :)
Gil
-Original Message-
From: Toke Eskildsen [mailto:t...@statsbiblioteket.dk]
Sent: 11 December 2013 12:02
To: solr-use
On Thu, 2013-12-12 at 02:46 +0100, Joel Bernstein wrote:
> Curious how many documents per shard you were planning?
350-500 million, optimized to a single segment as the data are not
changing.
> The number of documents per shard and field type will drive the amount
> of a RAM needed to sort and fa
Hi Gil,
I'd look at the number and type of fields you sort and facet on (this stuff
likes memory).
I'd keep in mind heaps over 32 GB use bigger pointers, so maybe more
smaller heaps are better than one big one.
You didn't mention the # of CPU cores, but keep that in mind when sharding.
When a que
Curious how many documents per shard you were planning? The number of
documents per shard and field type will drive the amount of a RAM needed to
sort and facet.
On Wed, Dec 11, 2013 at 7:02 AM, Toke Eskildsen wrote:
> On Tue, 2013-12-10 at 17:51 +0100, Hoggarth, Gil wrote:
> > We're probably go
On Tue, 2013-12-10 at 17:51 +0100, Hoggarth, Gil wrote:
> We're probably going to be building a Solr service to handle a dataset
> of ~60TB, which for our data and schema typically gives a Solr index
> size of 1/10th - i.e., 6TB. Given there's a general rule about the
> amount of hardware memory re
Shawn's right that if you're going to scale this big you'd be very well
served to spend time getting the index as small as possible. In my
experience if your searches require real-time random access reads (that is,
the entire index needs to be fast), you don't want to wait for HDD disk
reads.
Get
rag.org]
Sent: 10 December 2013 17:37
To: solr-user@lucene.apache.org
Subject: Re: Solr hardware memory question
On 12/10/2013 9:51 AM, Hoggarth, Gil wrote:
> We're probably going to be building a Solr service to handle a dataset
> of ~60TB, which for our data and schema typically gives
On 12/10/2013 9:51 AM, Hoggarth, Gil wrote:
> We're probably going to be building a Solr service to handle a dataset
> of ~60TB, which for our data and schema typically gives a Solr index
> size of 1/10th - i.e., 6TB. Given there's a general rule about the
> amount of hardware memory required shoul
10 matches
Mail list logo
