On 8/24/2018 5:17 AM, zhenyuan wei wrote:
Thanks again~ @Shawn @Jan Høydahl What is the recommend size of one shard？or how many docs per shard is recommend? My collection has 20 shard，each shard is 30~50GB。
There are no general rules. Every situation will be different, and advice that works for one situation can be completely wrong for another.
If there is no other information available, we figure that if you're going above 100 million docs per shard, that may be problematic. But for some situations, it works fine. If your documents are only 300 bytes, you might not have any problem going above 100 million.
You cannot tell Solr "cache this field, but do not cache this other field." The caches cannot be configured on a per-field basis like this.
The queryResultCache caches query results, if the result size is inside the window size. The filterCache caches filter results. The documentCache holds actual document contents, for the phase where Solr is retrieving documents for search results.
Each entry in the filterCache tends to be large. If you have one million documents in the shard, each filterCache entry for that shard will be 125000 bytes -- one eighth of the document count.
All I can do is resize filterCache，with 5 enumerate type，meybe 7.5GB x 5 = 37.5GB， and enable HDFS direcMemorySize as 20GB per node，using SSD for storage。 As a result, if query cache is 2 times of 37.5GB, 75GB， I should use 11 solr node，each solr jvm node using 6GB RAM， and each node enable 10GB hdfs directMemorySize。
These cache sizes that you're mentioning are impossible on a machine with only 32GB of memory, and definitely will not fit inside a heap size of 6GB.
Your machines are not big enough for the index you're creating, and I think you're probably going to need more machines.
Thanks, Shawn
