On 3/28/2018 1:06 AM, Bernd Fehling wrote:
Humongous Allocations are not genrally bad. Sure, the G1 part for humongous allocations is not that performant and takes time. But just try to limit humongous allocations and not to avoid it under all circumstances.
I was told by Oracle engineers that humongous allocations can ONLY be collected by a full GC. A full GC on my servers, with an 8GB heap, can take 10-15 seconds. This isn't a guess -- I've examined the GC logs. And that's a full stop-the-world pause for the entire collection. If I increase the region size, then filterCache entries are not tagged as humongous, and full GCs almost never happen.
Every single filterCache entry on my large shards was more than 2MB at the time I was experimenting. With an 8GB heap, the calculated G1 region size is 4MB. Anything that's half the region size or larger is considered a humongous allocation, so objects over 2MB get that tag.
Each server has three of those large shards, so when a new query goes into the filterCache, three large filterCache entries are created in one JVM. These days, each of those shards has 31 million docs, which means that a filterCache entry has almost doubled in size since then. It's almost 4 MB.
If one of those cores gets updated, multiple large objects become garbage, and because of cache autowarming (with autowarmCount set to 4), a few more get created. Each of our queries typically has several filter queries in it. Some of them don't really change, but a couple of them are highly variable from user to user. Solr burns through the heap pretty fast with those large filterCache entries. When it fills up with garbage objects tagged humongous, the only way to clear it out is a full GC, because the concurrent low-pause collectors are unable to collect humongous objects.
Because the region size tops out at 32MB, that means that any object over 16MB is always humongous. So for indexes with a very large number of documents per core (more than about 134 million), G1 is not a good choice. Unless the filterCache is completely disabled, which isn't normally recommended, G1 can't be tuned to avoid full GC with an index that size.
Thanks, Shawn
