What is the downside of configuring ramBufferSizeMB to be equal to 5GB ?
Is it only that the window of time for flush is larger, so recovery time will
be higher in case of a crash?
Thanks
Suresh
On 12/27/17, 1:34 PM, "Erick Erickson" <erickerick...@gmail.com> wrote:
You are probably hitting more and more background merging which will
slow things down. Your system looks to be severely undersized for this
scale.
One thing you can try (and I emphasize I haven't prototyped this) is
to increase your RamBufferSizeMB solrcofnig.xml setting significantly.
By default, Solr won't merge segments to greater than 5G, so
theoretically you could just set your ramBufferSizeMB to that figure
and avoid merging all together. Or you could try configuring the
NoMergePolicy in solrconfig.xml (but beware that you're going to
create a lot of segments unless you set the rambuffersize higher).
How this will affect your indexing throughput I frankly have no data.
You can see that with numbers like this, though, a 4G heap is much too
small.
Best,
Erick
On Wed, Dec 27, 2017 at 2:18 AM, Prasad Tendulkar
<pra...@cumulus-systems.com> wrote:
> Hello All,
>
> We have been building a Solr based solution to hold a large amount of
data (approx 4 TB/day or > 24 Billion documents per day). We are developing a
prototype on a small scale just to evaluate Solr performance gradually. Here is
our setup configuration.
>
> Solr cloud:
> node1: 16 GB RAM, 8 Core CPU, 1TB disk
> node2: 16 GB RAM, 8 Core CPU, 1TB disk
>
> Zookeeper is also installed on above 2 machines in cluster mode.
> Solr commit intervals: Soft commit 3 minutes, Hard commit 15 seconds
> Schema: Basic configuration. 5 fields indexed (out of one is
text_general), 6 fields stored.
> Collection: 12 shards (6 per node)
> Heap memory: 4 GB per node
> Disk cache: 12 GB per node
> Document is a syslog message.
>
> Documents are being ingested into Solr from different nodes. 12 SolrJ
clients ingest data into the Solr cloud.
>
> We are experiencing issues when we keep the setup running for long time
and after processing around 100 GB of index size (I.e. Around 600 Million
documents). Note that we are only indexing the data and not querying it. So
there should not be any query overhead. From the VM analysis we figured out
that over time the disk operations starts declining and so does the CPU, RAM
and Network usage of the Solr nodes. We concluded that Solr is unable to handle
one big collection due to index read/write overhead and most of the time it
ends up doing only the commit (evident in Solr logs). And because of that
indexing is getting hampered (?)
>
> So we thought of creating small sized collections instead of one big
collection anticipating the commit performance might improve. But eventually
the performance degrades even with that and we observe more or less similar
charts for CPU, memory, disk and network.
>
> To put forth some stats here are the number of documents processed every
hour
>
> 1St hour: 250 million
> 2nd hour: 250 million
> 3rd hour: 240 million
> 4th hour: 200 million
> .
> .
> 11th hour: 80 million
>
> Could you please help us identifying the root cause of degradation in the
performance? Are we doing something wrong with the Solr configuration or the
collections/sharding etc? Due to this performance degradation we are currently
stuck with Solr.
>
> Thank you very much in advance.
>
> Prasad Tendulkar
>
>
