Solr-8.1.0 uses much more memory
Comparing memory consumption (real, not virtual) of quiesent Solr v8.0 and prior with Solr v8.1.0 reveals the older versions use about 1.6GB on my systems but v8.1.0 uses 4.5 to 5+GB. Systems used are SUSE Linux, with Oracle JDK v1.8 and openjdk v10. This is a major memory consumption issue. I have seen no mention of it in the docs nor forums. Thanks, Joe D.
Re: Solr-8.1.0 uses much more memory
On 5/25/2019 9:40 AM, Joe Doupnik wrote: Comparing memory consumption (real, not virtual) of quiesent Solr v8.0 and prior with Solr v8.1.0 reveals the older versions use about 1.6GB on my systems but v8.1.0 uses 4.5 to 5+GB. Systems used are SUSE Linux, with Oracle JDK v1.8 and openjdk v10. This is a major memory consumption issue. I have seen no mention of it in the docs nor forums. If Solr is using 4 to 5 GB of memory on your system, it is only doing that because you told it that it was allowed to. If you run a Java program with a minimum heap that's smaller than the max heap, which Solr does not do by default, then what you will find is that Java *might* stay lower than the maximum for a while. But eventually it WILL allocate the entire maximum heap from the OS, plus some extra for Java itself to work with. Solr 8.0 and Solr 8.1 are not different from each other in this regard. Thanks, Shawn
Re: Solr-8.1.0 uses much more memory
On 26/05/2019 19:08, Shawn Heisey wrote: On 5/25/2019 9:40 AM, Joe Doupnik wrote: Comparing memory consumption (real, not virtual) of quiesent Solr v8.0 and prior with Solr v8.1.0 reveals the older versions use about 1.6GB on my systems but v8.1.0 uses 4.5 to 5+GB. Systems used are SUSE Linux, with Oracle JDK v1.8 and openjdk v10. This is a major memory consumption issue. I have seen no mention of it in the docs nor forums. If Solr is using 4 to 5 GB of memory on your system, it is only doing that because you told it that it was allowed to. If you run a Java program with a minimum heap that's smaller than the max heap, which Solr does not do by default, then what you will find is that Java *might* stay lower than the maximum for a while. But eventually it WILL allocate the entire maximum heap from the OS, plus some extra for Java itself to work with. Solr 8.0 and Solr 8.1 are not different from each other in this regard. Thanks, Shawn Not to be argumentative, prior to Solr v8.1 quiesent resident memory remained at about the 1.6GB level, and during active indexing it could exceed 3.5GB. With the same configuration settings Solr v8.1 changes that to use _a lot_ more memory. Thus something significant has changed with Solr v8.1 when compared to its predecessors. The question is what, and what can we do about it. I am not about to enter a guessing game with Solr and Java and its heap usage. That is far to complex to hope to win. Thus, something changed, for the worse here in the field, and I do not know what. Thanks, Joe D.
Re: Solr-8.1.0 uses much more memory
Different garbage collector configuration? It does not mean that Solr uses more memory if it is occupied - it could also mean that the JVM just kept it reserved for future memory needs. > Am 25.05.2019 um 17:40 schrieb Joe Doupnik : > > Comparing memory consumption (real, not virtual) of quiesent Solr v8.0 > and prior with Solr v8.1.0 reveals the older versions use about 1.6GB on my > systems but v8.1.0 uses 4.5 to 5+GB. Systems used are SUSE Linux, with Oracle > JDK v1.8 and openjdk v10. This is a major memory consumption issue. I have > seen no mention of it in the docs nor forums. > Thanks, > Joe D.
Re: Solr-8.1.0 uses much more memory
On 26/05/2019 19:15, Joe Doupnik wrote: On 26/05/2019 19:08, Shawn Heisey wrote: On 5/25/2019 9:40 AM, Joe Doupnik wrote: Comparing memory consumption (real, not virtual) of quiesent Solr v8.0 and prior with Solr v8.1.0 reveals the older versions use about 1.6GB on my systems but v8.1.0 uses 4.5 to 5+GB. Systems used are SUSE Linux, with Oracle JDK v1.8 and openjdk v10. This is a major memory consumption issue. I have seen no mention of it in the docs nor forums. If Solr is using 4 to 5 GB of memory on your system, it is only doing that because you told it that it was allowed to. If you run a Java program with a minimum heap that's smaller than the max heap, which Solr does not do by default, then what you will find is that Java *might* stay lower than the maximum for a while. But eventually it WILL allocate the entire maximum heap from the OS, plus some extra for Java itself to work with. Solr 8.0 and Solr 8.1 are not different from each other in this regard. Thanks, Shawn Not to be argumentative, prior to Solr v8.1 quiesent resident memory remained at about the 1.6GB level, and during active indexing it could exceed 3.5GB. With the same configuration settings Solr v8.1 changes that to use _a lot_ more memory. Thus something significant has changed with Solr v8.1 when compared to its predecessors. The question is what, and what can we do about it. I am not about to enter a guessing game with Solr and Java and its heap usage. That is far to complex to hope to win. Thus, something changed, for the worse here in the field, and I do not know what. Thanks, Joe D. --- If I were forced to guess about this situation it woud be to flag an item mentioned vaguely in passing: the garbage collector. How to return it to status quo ante is not known here. Presumably such a step would be covered in the yet to appear documentation for Solr v8.1 To add a little more to the story. Memory remained at the 1.6GB level except when doing heavy indexing. To "adjust" Solr so that it always consumes too much, as at present, is not acceptable, nor is acceptable risking trouble by setting an upper limit down to say 1.6GB and thence cause indexing to fail. We see the dilemna. Expert assistance is needed to resolve this. Thanks, Joe D.
Re: Solr-8.1.0 uses much more memory
I think this is also a very risky memory strategy. What happens if you Index and query at the same time etc. maybe it is more worth to provide as much memory as for concurrent operations are needed. This includes JVM memory but also the disk caches. > Am 26.05.2019 um 20:38 schrieb Joe Doupnik : > >> On 26/05/2019 19:15, Joe Doupnik wrote: >>> On 26/05/2019 19:08, Shawn Heisey wrote: On 5/25/2019 9:40 AM, Joe Doupnik wrote: Comparing memory consumption (real, not virtual) of quiesent Solr v8.0 and prior with Solr v8.1.0 reveals the older versions use about 1.6GB on my systems but v8.1.0 uses 4.5 to 5+GB. Systems used are SUSE Linux, with Oracle JDK v1.8 and openjdk v10. This is a major memory consumption issue. I have seen no mention of it in the docs nor forums. >>> >>> If Solr is using 4 to 5 GB of memory on your system, it is only doing that >>> because you told it that it was allowed to. >>> >>> If you run a Java program with a minimum heap that's smaller than the max >>> heap, which Solr does not do by default, then what you will find is that >>> Java *might* stay lower than the maximum for a while. But eventually it >>> WILL allocate the entire maximum heap from the OS, plus some extra for Java >>> itself to work with. Solr 8.0 and Solr 8.1 are not different from each >>> other in this regard. >>> >>> Thanks, >>> Shawn >> >> Not to be argumentative, prior to Solr v8.1 quiesent resident memory >> remained at about the 1.6GB level, and during active indexing it could >> exceed 3.5GB. With the same configuration settings Solr v8.1 changes that to >> use _a lot_ more memory. Thus something significant has changed with Solr >> v8.1 when compared to its predecessors. The question is what, and what can >> we do about it. >> I am not about to enter a guessing game with Solr and Java and its heap >> usage. That is far to complex to hope to win. >> Thus, something changed, for the worse here in the field, and I do not >> know what. >> Thanks, >> Joe D. > --- > If I were forced to guess about this situation it woud be to flag an item > mentioned vaguely in passing: the garbage collector. How to return it to > status quo ante is not known here. Presumably such a step would be covered in > the yet to appear documentation for Solr v8.1 > To add a little more to the story. Memory remained at the 1.6GB level > except when doing heavy indexing. To "adjust" Solr so that it always consumes > too much, as at present, is not acceptable, nor is acceptable risking trouble > by setting an upper limit down to say 1.6GB and thence cause indexing to fail. > We see the dilemna. Expert assistance is needed to resolve this. > Thanks, > Joe D.
Re: Solr-8.1.0 uses much more memory
On 26/05/2019 19:38, Jörn Franke wrote: Different garbage collector configuration? It does not mean that Solr uses more memory if it is occupied - it could also mean that the JVM just kept it reserved for future memory needs. Am 25.05.2019 um 17:40 schrieb Joe Doupnik : Comparing memory consumption (real, not virtual) of quiesent Solr v8.0 and prior with Solr v8.1.0 reveals the older versions use about 1.6GB on my systems but v8.1.0 uses 4.5 to 5+GB. Systems used are SUSE Linux, with Oracle JDK v1.8 and openjdk v10. This is a major memory consumption issue. I have seen no mention of it in the docs nor forums. Thanks, Joe D. --- The garbage collector was on my mind as well (in a msg sent just before yours). These numbers are easy to verify, just by using "top". They say allocated, meaning Java owns it, no matter what Java does with it. Java does not own the machine; there are other useful activities to tend as well. Let's find the problem and cure it. Thanks, Joe D.
Re: Solr-8.1.0 uses much more memory
I do queries while indexing, have done so for a long time, without difficulty nor memory usage spikes from dual use. The system has been designed to support that. Again, one may look at the numbers using "top" or similar. Try Solr v8.0 and 8.1 to see the difference which I experience here. For reference, the only memory adjustables set in my configuration is in the Solr startup script solr.in.sh saying add "-Xss1024k" in the SOLR_OPTS list and setting SOLR_HEAP="4024m". Thanks, Joe D. On 26/05/2019 19:43, Jörn Franke wrote: I think this is also a very risky memory strategy. What happens if you Index and query at the same time etc. maybe it is more worth to provide as much memory as for concurrent operations are needed. This includes JVM memory but also the disk caches. Am 26.05.2019 um 20:38 schrieb Joe Doupnik : On 26/05/2019 19:15, Joe Doupnik wrote: On 26/05/2019 19:08, Shawn Heisey wrote: On 5/25/2019 9:40 AM, Joe Doupnik wrote: Comparing memory consumption (real, not virtual) of quiesent Solr v8.0 and prior with Solr v8.1.0 reveals the older versions use about 1.6GB on my systems but v8.1.0 uses 4.5 to 5+GB. Systems used are SUSE Linux, with Oracle JDK v1.8 and openjdk v10. This is a major memory consumption issue. I have seen no mention of it in the docs nor forums. If Solr is using 4 to 5 GB of memory on your system, it is only doing that because you told it that it was allowed to. If you run a Java program with a minimum heap that's smaller than the max heap, which Solr does not do by default, then what you will find is that Java *might* stay lower than the maximum for a while. But eventually it WILL allocate the entire maximum heap from the OS, plus some extra for Java itself to work with. Solr 8.0 and Solr 8.1 are not different from each other in this regard. Thanks, Shawn Not to be argumentative, prior to Solr v8.1 quiesent resident memory remained at about the 1.6GB level, and during active indexing it could exceed 3.5GB. With the same configuration settings Solr v8.1 changes that to use _a lot_ more memory. Thus something significant has changed with Solr v8.1 when compared to its predecessors. The question is what, and what can we do about it. I am not about to enter a guessing game with Solr and Java and its heap usage. That is far to complex to hope to win. Thus, something changed, for the worse here in the field, and I do not know what. Thanks, Joe D. --- If I were forced to guess about this situation it woud be to flag an item mentioned vaguely in passing: the garbage collector. How to return it to status quo ante is not known here. Presumably such a step would be covered in the yet to appear documentation for Solr v8.1 To add a little more to the story. Memory remained at the 1.6GB level except when doing heavy indexing. To "adjust" Solr so that it always consumes too much, as at present, is not acceptable, nor is acceptable risking trouble by setting an upper limit down to say 1.6GB and thence cause indexing to fail. We see the dilemna. Expert assistance is needed to resolve this. Thanks, Joe D.
Re: Solr-8.1.0 uses much more memory
On 5/26/2019 12:52 PM, Joe Doupnik wrote: I do queries while indexing, have done so for a long time, without difficulty nor memory usage spikes from dual use. The system has been designed to support that. Again, one may look at the numbers using "top" or similar. Try Solr v8.0 and 8.1 to see the difference which I experience here. For reference, the only memory adjustables set in my configuration is in the Solr startup script solr.in.sh saying add "-Xss1024k" in the SOLR_OPTS list and setting SOLR_HEAP="4024m". There is one significant difference between 8.0 and 8.1 in the realm of memory management -- we have switched from the CMS garbage collector to the G1 collector. So the way that Java manages the heap has changed. This was done because the CMS collector is slated for removal from Java. https://issues.apache.org/jira/browse/SOLR-13394 Java is unlike other programs in one respect -- once it allocates heap from the OS, it never gives it back. This behavior has given Java an undeserved reputation as a memory hog ... but in fact Java's overall memory usage can be very easily limited ... an option that many other programs do NOT have. In your configuration, you set the max heap to a little less than 4GB. You have to expect that it *WILL* use that memory. By using the SOLR_HEAP variable, you have instructed Solr's startup script to use the same setting for the minimum heap as well as the maximum heap. This is the design intent. If you want to know how much heap is being used, you can't ask the operating system, which means tools like top. You have to ask Java. And you will have to look at a long-term graph, finding the low points. An instananeous look at Java's heap usage could show you that the whole heap is allocated ... but a significant part of that allocation could be garbage, which becomes available once the garbage is collected. Thanks, Shawn
Re: Solr-8.1.0 uses much more memory
I'm not sure this issue applies in this situation but it's worth taking a look at: https://issues.apache.org/jira/browse/SOLR-12833?focusedCommentId=16807868&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16807868 Although the memory issue in the ticket involves different versions than I think are being discussed. It's good to understand that this issue exists and that it's resolved going forward. Also because the way that this issue is attached to the original ticket that caused the bug, rather than a new bug report, it's very hard to know that this problem actually existed. Joel Bernstein http://joelsolr.blogspot.com/ On Sun, May 26, 2019 at 3:30 PM Shawn Heisey wrote: > On 5/26/2019 12:52 PM, Joe Doupnik wrote: > > I do queries while indexing, have done so for a long time, without > > difficulty nor memory usage spikes from dual use. The system has been > > designed to support that. > > Again, one may look at the numbers using "top" or similar. Try Solr > > v8.0 and 8.1 to see the difference which I experience here. For > > reference, the only memory adjustables set in my configuration is in the > > Solr startup script solr.in.sh saying add "-Xss1024k" in the SOLR_OPTS > > list and setting SOLR_HEAP="4024m". > > There is one significant difference between 8.0 and 8.1 in the realm of > memory management -- we have switched from the CMS garbage collector to > the G1 collector. So the way that Java manages the heap has changed. > This was done because the CMS collector is slated for removal from Java. > > https://issues.apache.org/jira/browse/SOLR-13394 > > Java is unlike other programs in one respect -- once it allocates heap > from the OS, it never gives it back. This behavior has given Java an > undeserved reputation as a memory hog ... but in fact Java's overall > memory usage can be very easily limited ... an option that many other > programs do NOT have. > > In your configuration, you set the max heap to a little less than 4GB. > You have to expect that it *WILL* use that memory. By using the > SOLR_HEAP variable, you have instructed Solr's startup script to use the > same setting for the minimum heap as well as the maximum heap. This is > the design intent. > > If you want to know how much heap is being used, you can't ask the > operating system, which means tools like top. You have to ask Java. > And you will have to look at a long-term graph, finding the low points. > An instananeous look at Java's heap usage could show you that the whole > heap is allocated ... but a significant part of that allocation could be > garbage, which becomes available once the garbage is collected. > > Thanks, > Shawn >
