-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hendrik,
On 10/12/18 02:36, Hendrik Haddorp wrote: > Those constraints can be easily set if you are using Docker. The > problem is however that at least up to Oracle Java 8, and I believe > quite a bit further, the JVM is not at all aware about those > limits. That's why when running Solr in Docker you really need to > make sure that you set the memory limits lower. I usually set the > heap and metaspace size. How you set them depends again a bit on > your Solr configuration. I prefer the JVM to crash due to memory > limits rather then the Linux OOM Killer killing the JVM as the > OutOfMemoryError from the JVM does at least state what memory was > out. Limiting the native memory used by attempting to limit the heap is not actually limiting the native memory used. It's just an attempt to do so. If you limit the native memory using OS limits (or, using Docker, simply make it look like there is less system memory) then you haven't actually achieved anything. You could have done that simply by lowering heap values and avoided the complexity of Docker, etc. - -chris > On 11.10.2018 16:45, Christopher Schultz wrote: Shawn, > > On 10/11/18 12:54 AM, Shawn Heisey wrote: >>>> On 10/10/2018 10:08 PM, Sourav Moitra wrote: >>>>> We have a Solr server with 8gb of memory. We are using solr >>>>> in cloud mode, solr version is 7.5, Java version is Oracle >>>>> Java 9 and settings for Xmx and Xms value is 2g but we are >>>>> observing that the RAM getting used to 98% when doing >>>>> indexing. >>>>> >>>>> How can I ensure that SolrCloud doesn't use more than N GB >>>>> of memory ? >>>> Where precisely are you seeing the 98% usage? It is >>>> completely normal for a modern operating system to report >>>> that almost all the system memory is in use, at least after >>>> the system has been shuffling a lot of data. All modern >>>> operating systems will use memory that has not been >>>> specifically allocated to programs for disk caching purposes, >>>> and system information tools will generally indicate that >>>> this memory is in use, even though it can be instantly >>>> claimed by any program that requests it. >>>> >>>> https://en.wikipedia.org/wiki/Page_cache >>>> >>>> If you tell a Java program that it is limited to a 2GB heap, >>>> then that program will never use more than 2GB, plus a little >>>> extra for the java runtime itself. I cannot give you an >>>> exact figure for that little bit extra. But every bit of >>>> data on disk that Solr accesses will end up (at least >>>> temporarily) in the operating system's disk cache -- using >>>> that unallocated memory. >>>> >>>> https://wiki.apache.org/solr/SolrPerformanceProblems#RAM > To be fair, the JVM can use *much more* memory than you have > specified for your Java heap. It's just that the Java heap itself > wont exceed those values. > > The JVM uses quite a bit of native memory which isn't counted in > the Java heap. There is only one way I know of to control that, and > it's to set a process-limit at the OS level on the amount of > memory allowed. I'm not sure how sensitive to those limits the JVM > actually is, so attempting to artificially constrain the JVM might > end up with a native OOM crash. > > -chris > -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvA4O0ACgkQHPApP6U8 pFiCHg/+P+/yoSrvMd2uMyDK16nMCOIdxAL1gdS++DqS+qPmch1BHJTA9nuHybF4 j6WElpCI7Q3HP/sgsGE8kHE6Kg+DFJNz7mGJqgXjnSkm90LzETRFMqa959fTgBo6 SILD4n4LnZI844VoaKb2gIVibr804hloxX5UDe0XYFp3EtcVi4QMC5Q2ovn8+RoJ S/LJx/VQi3AqtcCaEYAAKpYrKxO3OkoIKnN+oC55ag/16zh9StT2TUI03bBslcxn PkS5zdsSmsS7NydSR4Gn4C7wAGyL3hGoU6pD+GhvYE9EF29KxHXFSIe2FJQ6mdRf ikZvm17U8OFNwqlB4OOLziGvOkcmIgtqchnhUm80Qwtn0ZMbql2zwlIhOSPWbuPL lq3F09p1QBqPjbxJdrcmpoSFH8jvmIPdrPOl3BbPEmDzNdnF03sEGP5gDyJ9/INB AD/QhqvQEKUtMBPX+1/9dxOm+JyUDlARZQ7p4k1BeFjl2BI8imLUK/c6JlWJ757G QWk+0Ff3R02va+ITWNvGs5C1uOnu2g58eqAggREPWXmXAj9nqJ5EyPkNAaGJBheo NasGNSXVnjN+hk4QlMTAJ3C5u0Q5lW3HCOXj8Mufo7LE8M96OjRkM09o87NG9sGT EdX7V8Ypw758Jt9xcms6U9tC2TqekJ9AYu+VLsoGa4OZgy5hfDk= =Sq+f -----END PGP SIGNATURE-----
