On 2/7/2018 8:08 AM, Shawn Heisey wrote:
If your queries are producing the correct results,
then I will tell you that the "summary" part of your query example is
quite possibly completely unnecessary
After further thought, I have concluded that this part of what I said is
probably completely w
On 2/7/2018 5:20 AM, Maulin Rathod wrote:
Further analyzing issue we found that asking for too many rows (e.g.
rows=1000) can cause full GC problem as mentioned in below link.
This is because when you ask for 10 million rows, Solr allocates a
memory structure capable of storing informatio
t: 31 January 2018 22:47
To: solr-user
Subject: Re: Long GC Pauses
Just to double check, when you san you're seeing 60-200 sec GC pauses are you
looking at the GC logs (or using some kind of monitor) or is that the time it
takes the query to respond to the client? Because a single
To: solr-user
Subject: Re: Long GC Pauses
Just to double check, when you san you're seeing 60-200 sec GC pauses are you
looking at the GC logs (or using some kind of monitor) or is that the time it
takes the query to respond to the client? Because a single GC pause that long
on 40G is unu
Hey Maulin,
I hope you are using some tools to look at your gc.log file (There are
couple available online) or grepping for pauses.
Do you mind sharing your G1GC settings and some screenshots from your
gc.log analyzer's output ?
-SG
On Wed, Jan 31, 2018 at 9:16 AM, Erick Erickson
wrote:
> Jus
Just to double check, when you san you're seeing 60-200 sec GC pauses
are you looking at the GC logs (or using some kind of monitor) or is
that the time it takes the query to respond to the client? Because a
single GC pause that long on 40G is unusual no matter what. Another
take on Jason's questi
Hi Maulin,
To clarify, when you said "...allocated 40 GB RAM to each shard." above,
I'm going to assume you meant "to each node" instead. If you actually did
mean "to each shard" above, please correct me and anyone who chimes in
afterward.
Firstly, it's really hard to even take guesses about pot
Chetas Joshi wrote:
> Thanks for the insights into the memory requirements. Looks like cursor
> approach is going to require a lot of memory for millions of documents.
Sorry, that is a premature conclusion from your observations.
> If I run a query that returns only 500K documents still keeping
On 4/13/2017 11:51 AM, Chetas Joshi wrote:
> Thanks for the insights into the memory requirements. Looks like cursor
> approach is going to require a lot of memory for millions of documents.
> If I run a query that returns only 500K documents still keeping 100K docs
> per page, I don't see long GC
Hi Shawn,
Thanks for the insights into the memory requirements. Looks like cursor
approach is going to require a lot of memory for millions of documents.
If I run a query that returns only 500K documents still keeping 100K docs
per page, I don't see long GC pauses. So it is not really the number o
You're missing the point of my comment. Since they already are
docValues, you can use the /export functionality to get the results
back as a _stream_ and avoid all of the overhead of the aggregator
node doing a merge sort and all of that.
You'll have to do this from SolrJ, but see CloudSolrStream.
On 4/12/2017 5:19 PM, Chetas Joshi wrote:
> I am getting back 100K results per page.
> The fields have docValues enabled and I am getting sorted results based on
> "id" and 2 more fields (String: 32 Bytes and Long: 8 Bytes).
>
> I have a solr Cloud of 80 nodes. There will be one shard that will ge
I am getting back 100K results per page.
The fields have docValues enabled and I am getting sorted results based on
"id" and 2 more fields (String: 32 Bytes and Long: 8 Bytes).
I have a solr Cloud of 80 nodes. There will be one shard that will get top
100K docs from each shard and apply merge sort
Oh my. Returning 100K rows per request is usually poor practice.
One hopes these are very tiny docs.
But this may well be an "XY" problem. What kinds of information
are you returning in your docs and could they all be docValues
types? In which case you would be waaay far ahead by using
the various
I am running a query that returns 10 MM docs in total and the number of
rows per page is 100K.
On Wed, Apr 12, 2017 at 12:53 PM, Mikhail Khludnev wrote:
> And what is the rows parameter?
>
> 12 апр. 2017 г. 21:32 пользователь "Chetas Joshi"
> написал:
>
> > Thanks for your response Shawn and Wu
And what is the rows parameter?
12 апр. 2017 г. 21:32 пользователь "Chetas Joshi"
написал:
> Thanks for your response Shawn and Wunder.
>
> Hi Shawn,
>
> Here is the system config:
>
> Total system memory = 512 GB
> each server handles two 500 MB cores
> Number of solr docs per 500 MB core = 200
Thanks for your response Shawn and Wunder.
Hi Shawn,
Here is the system config:
Total system memory = 512 GB
each server handles two 500 MB cores
Number of solr docs per 500 MB core = 200 MM
The average heap usage is around 4-6 GB. When the read starts using the
Cursor approach, the heap usage
JVM version? We’re running v8 update 121 with the G1 collector and it is
working really well. We also have an 8GB heap.
Graph your heap usage. You’ll see a sawtooth shape, where it grows, then there
is a major GC. The maximum of the base of the sawtooth is the working set of
heap that your Solr
On 4/11/2017 2:56 PM, Chetas Joshi wrote:
> I am using Solr (5.5.0) on HDFS. SolrCloud of 80 nodes. Sold collection
> with number of shards = 80 and replication Factor=2
>
> Sold JVM heap size = 20 GB
> solr.hdfs.blockcache.enabled = true
> solr.hdfs.blockcache.direct.memory.allocation = true
> Max
19 matches
Mail list logo
