Toke Eskildsen [t...@statsbiblioteket.dk] wrote:
[Toke: SSDs with 2.7TB of index on a 256GB machine]
> tl;dr: for small result sets (< 1M hits) on unwarmed searches with
> simple queries, response time is below 100ms. If we enable faceting with
> plain Solr, this jumps to about 1 second.
> I did
Earlier I used to index with HtttpPost Mechanism only, making each post
size specific to 2Mb to 20Mb that was going fine, but we had a suspect that
instead of indexing through network call(which ofcourse results in latency
due to network delays and http protocol) if we can index Offline by just
wri
On Fri, 2014-06-06 at 14:05 +0200, Vineet Mishra wrote:
> Could you state what indexing mechanism are you using, as I started
> with EmbeddedSolrServer but it was pretty slow after a few GB(~30+) of
> indexing.
I suspect that is due to too-frequent commits, too small heap or
something third, unre
Hi Toke,
That was Spectacular, really great to hear that you have already indexed
2.7TB+ data to your server and still the query response time is under ms or
a few seconds for such a huge dataset.
Could you state what indexing mechanism are you using, as I started with
EmbeddedSolrServer but it wa
On Fri, 2014-06-06 at 12:32 +0200, Vineet Mishra wrote:
> *Does that mean for querying smoothly we need to have memory atleast equal
> or greater to the size of index?
If you absolutely, positively have to reduce latency as much as
possible, then yes. With an estimated index size of 2TB, I would g
hing
> you need to know.
>
>
> -- Jack Krupansky
>
> -Original Message- From: Vineet Mishra
> Sent: Wednesday, June 4, 2014 2:45 AM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr maximum Optimal Index Size per Shard
>
>
> Thanks all for your response.
>
Hi Shawn,
Thanks for your response, wanted to clarify a few things.
*Does that mean for querying smoothly we need to have memory atleast equal
or greater to the size of index? As in my case the index size will be very
heavy(~2TB) and practically speaking that amount of memory is not possible.
Eve
ation will tell you everything you
need to know.
-- Jack Krupansky
-Original Message-
From: Vineet Mishra
Sent: Wednesday, June 4, 2014 2:45 AM
To: solr-user@lucene.apache.org
Subject: Re: Solr maximum Optimal Index Size per Shard
Thanks all for your response.
I presume this conver
On 6/4/2014 12:45 AM, Vineet Mishra wrote:
> Thanks all for your response.
> I presume this conversation concludes that indexing around 1Billion
> documents per shard won't be a problem, as I have 10 Billion docs to index,
> so approx 10 shards with 1 Billion each should be fine with it and how
> a
Thanks all for your response.
I presume this conversation concludes that indexing around 1Billion
documents per shard won't be a problem, as I have 10 Billion docs to index,
so approx 10 shards with 1 Billion each should be fine with it and how
about Memory, what size of RAM should be fine for this
mark.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Solr-maximum-Optimal-Index-Size-per-Shard-tp4139565p4139698.html
Sent from the Solr - User mailing list archive at Nabble.com.
On 6/3/2014 1:47 PM, Jack Krupansky wrote:
> Anybody care to forecast when hardware will catch up with Solr and we
> can routinely look forward to newbies complaining that they indexed
> "some" data and after only 10 minutes they hit this weird 2G document
> count limit?
I would speculate that Luc
isey
Sent: Tuesday, June 3, 2014 3:34 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr maximum Optimal Index Size per Shard
On 6/3/2014 12:54 PM, Jack Krupansky wrote:
How much free system memory do you have for the OS to cache file
system data? If your entire index fits in system memory opera
On 6/3/2014 12:54 PM, Jack Krupansky wrote:
> How much free system memory do you have for the OS to cache file
> system data? If your entire index fits in system memory operations
> will be fast, but as your index grows beyond the space the OS can use
> to cache the data, performance will decline.
How much free system memory do you have for the OS to cache file system
data? If your entire index fits in system memory operations will be fast,
but as your index grows beyond the space the OS can use to cache the data,
performance will decline.
But there's no hard limit in Solr per se.
-- J
15 matches
Mail list logo
