firstSearcher per SolrCore or JVM?
I am using external file fields with larger external files and I noticed Solr Core Reload loads external files twice: firstSearcher and nextSearcher event. Does it mean the Core Reload triggers both events? What is the benefit/reason of triggering both events at the same time? I see this on V. 4.10.4 and V. 6.2.0 both. Thanks,
Re: firstSearcher per SolrCore or JVM?
Thanks Eric. FirstSearcher and newSearcher events open with two separate searchers. For the external file field case at least, the cache created with the firstSearcher is not being used after the newSearcher creates another cache (with same values) I believe the warming is also per searcher. So, I am still wondering why two searcher is running during the core reload. (since the cache with the firstSearcher is not used much) On V 4.10.4, it seems the main (current) searcher is closed after newSearcher event. So, all cache is generated with a new searcher before accepting a query with the new searcher. On V. 6.2.0, the main (current) searcher is closed first and then the newSearcher event is triggered. So, the first HTTP request with an external file field needs to wait until the cache is created with the newSearcher. In fact, two threads are running and I am not yet sure how they are synchronised (or not synchronized) Now sure, if I can attache an image here, but this is response time on V. 6.2.10 On Mon, Oct 3, 2016 at 9:00 PM, Erick Erickson wrote: > firstSearcher and newSeacher are definitely per core, they have to be > since they > are intended to warm searchers and searchers are per core. > > I don't particularly see the benefit of firing them both either. Not > sure which one makes > the most sense though. > > Best, > Erick > > On Mon, Oct 3, 2016 at 7:10 PM, Jihwan Kim wrote: > > I am using external file fields with larger external files and I noticed > > Solr Core Reload loads external files twice: firstSearcher and > nextSearcher > > event. > > > > Does it mean the Core Reload triggers both events? What is the > > benefit/reason of triggering both events at the same time? I see this > on > > V. 4.10.4 and V. 6.2.0 both. > > > > Thanks, >
Re: running solr 6.x in Eclipse for debugging
I read your first reference and run the following command on the Solr_Installed Dir. I am using v. 6.2.0 and 4.10.4. both works. bin/solr start -f -a "-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=7666" On Tue, Oct 4, 2016 at 5:26 PM, John Bickerstaff wrote: > All, > > I've found some hints, but can't get Solr running in debug mode in eclipse > with the instructions > > Here: > http://opensourceconnections.com/blog/2013/04/13/how-to- > debug-solr-with-eclipse/ > > or Here: http://wiki.apache.org/solr/HowToConfigureEclipse > > When I run this command from the wiki : java -Xdebug > -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=7666 -jar > start.jar > > I get an error: WARNING: Nothing to start, exiting ... > > I can't start the jar file "normally" either and it turns out to be a > symlink. > > I'm thinking that things are different now given that the way to start Solr > is this way: > > bin/solr start -e cloud -noprompt > > Can anyone point me at documentation or drop a hint? >
Enhancement on a getFloats method of FileFloatSource.java ?
I would like to ask the Solr organization an enhancement request.
The FileFloatSource creates a cache value from an external file when a Core
is reloaded and/or a new searcher is opened. Nevertheless, the external
files can be changed less frequently.
With a larger document and larger external files, it consumes lots of CPU
and memory. Also, it makes a Core reloading process longer.
Until the getFloats method of the FileFloatSource is completed and replace
the cache value in the floatCache --> readerCache, two larger float arrays
are in the memory. When the external file was not changed, these two
larger float array contains same values.
Current format of the external field file is this:
Id=float_value
Id=float_value
Id=float_value
: :
If we support this kind format, we can only create the large array
object(s) when a version of the file is changed.
Version (such as system current time as an example)
Id=float_value
Id=float_value
Id=float_value
: :
When the version was not changed, we can still use the cached array without
creating a new one.
Brief pseudo code, which should be added to the getFloats method, is
something like this.
String currentVersion = r.readLine();
if(latestVerstion != null && latestVerstion.equals(currentVersion)){
Object cacheVal = latestCache.get(ffs.field.getName());
if(null != cacheVal){
return (float[])cacheVal;
}
}
//Create float array after the version check.
vals = new float[reader.maxDoc()];
if (ffs.defVal != 0) {
Arrays.fill(vals, ffs.defVal);
}
latestVerstion = currentVersion;
latestCache.put(ffs.field.getName(), vals);
//This is existing codes
for (String line; (line=r.readLine())!=null;) {
int delimIndex = line.lastIndexOf(delimiter);
if (delimIndex < 0) continue;
int endIndex = line.length();
*How can I file the enhancement request?*
Thanks.
Re: Enhancement on a getFloats method of FileFloatSource.java ?
"The array is indexed by internal lucene docid," --> If I understood correctly, it is done inside the for loop that I briefly showed. In the following code I used, the 'vals' points to an array object and the latestCache puts the reference of the same array object. Then, the index decision and assigning a value in the array object is happening inside the for loop in the getFloats. So, the latestCache still hold same array object after the index decision is made along with value assignment from a value in the external file. During a QueryHandler (If I understood it correctly), I noticed that another array per segment is created (with a size 20 when I looked at) and this array is used query results & values in the float array by calling the public Object get(IndexReader reader, Object key) method. This get method uses the same float array created by the getFloats. vals = new float[reader.maxDoc()]; latestCache.put(ffs.field.getName(), vals); Am I missing something? Any feedback will be helpful to understand the Solr better. Thank you, Jihwan On Tue, Oct 4, 2016 at 8:35 PM, Yonik Seeley wrote: > On Tue, Oct 4, 2016 at 10:09 PM, Jihwan Kim wrote: > > I would like to ask the Solr organization an enhancement request. > > > > The FileFloatSource creates a cache value from an external file when a > Core > > is reloaded and/or a new searcher is opened. Nevertheless, the external > > files can be changed less frequently. > > > [...] > > When the version was not changed, we can still use the cached array > without > > creating a new one. > > The array is indexed by internal lucene docid, which can change across > different views of the index (a commit). > > What we can do is cache per segment though. When this class was > originally written, we didn't have access to individual segments. > > -Yonik >
Re: Enhancement on a getFloats method of FileFloatSource.java ?
Hi Yonik, I thought about your comment and I might understand what you were saying. The for loop in the getFloats method assign a different index of the array whenever a new segment is created/updated. You are saying that is why we cannot cache the float array as I suggested. Am I understood correctly? Thanks. On Tue, Oct 4, 2016 at 8:59 PM, Jihwan Kim wrote: > "The array is indexed by internal lucene docid," --> If I understood > correctly, it is done inside the for loop that I briefly showed. > > In the following code I used, the 'vals' points to an array object and the > latestCache puts the reference of the same array object. Then, the index > decision and assigning a value in the array object is happening inside the > for loop in the getFloats. So, the latestCache still hold same array > object after the index decision is made along with value assignment from a > value in the external file. > > During a QueryHandler (If I understood it correctly), I noticed that > another array per segment is created (with a size 20 when I looked at) and > this array is used query results & values in the float array by calling the > public Object get(IndexReader reader, Object key) method. This get > method uses the same float array created by the getFloats. > > vals = new float[reader.maxDoc()]; > > latestCache.put(ffs.field.getName(), vals); > > Am I missing something? Any feedback will be helpful to understand the > Solr better. > > Thank you, > > Jihwan > > On Tue, Oct 4, 2016 at 8:35 PM, Yonik Seeley wrote: > >> On Tue, Oct 4, 2016 at 10:09 PM, Jihwan Kim wrote: >> > I would like to ask the Solr organization an enhancement request. >> > >> > The FileFloatSource creates a cache value from an external file when a >> Core >> > is reloaded and/or a new searcher is opened. Nevertheless, the external >> > files can be changed less frequently. >> > >> [...] >> > When the version was not changed, we can still use the cached array >> without >> > creating a new one. >> >> The array is indexed by internal lucene docid, which can change across >> different views of the index (a commit). >> >> What we can do is cache per segment though. When this class was >> originally written, we didn't have access to individual segments. >> >> -Yonik >> > >
Re: Enhancement on a getFloats method of FileFloatSource.java ?
Got it! Thanks a lot! On Oct 4, 2016 9:29 PM, "Yonik Seeley" wrote: > On Tue, Oct 4, 2016 at 11:23 PM, Jihwan Kim wrote: > > Hi Yonik, > > I thought about your comment and I might understand what you were saying. > > The for loop in the getFloats method assign a different index of the > array > > whenever a new segment is created/updated. You are saying that is why we > > cannot cache the float array as I suggested. Am I understood correctly? > > Yes. > > For example, id="mydocument1" may map to lucene_docid 7 > and then after a new searcher is opened that same id may map to > lucene_docid 22 > > -Yonik >
Memory Issue with SnapPuller
Hi,
We are using Solr 4.10.4 and experiencing out of memory exception. It
seems the problem is cause by the following code & scenario.
This is the last part of a fetchLastIndex method in SnapPuller.java
// we must reload the core after we open the IW back up
if (reloadCore) {
reloadCore();
}
if (successfulInstall) {
if (isFullCopyNeeded) {
// let the system know we are changing dir's and the old one
// may be closed
if (indexDir != null) {
LOG.info("removing old index directory " + indexDir);
core.getDirectoryFactory().doneWithDirectory(indexDir);
core.getDirectoryFactory().remove(indexDir);
}
}
if (isFullCopyNeeded) {
solrCore.getUpdateHandler().newIndexWriter(isFullCopyNeeded);
}
openNewSearcherAndUpdateCommitPoint(isFullCopyNeeded);
}
Inside the reloadCore, it create a new core, register it, and try to close
the current/old core. When the closing old core process goes normal, it
throws an exception "SnapPull failed :org.apache.solr.common.SolrException:
Index fetch failed Caused by java.lang.RuntimeException: Interrupted while
waiting for core reload to finish Caused by Caused by:
java.lang.InterruptedException."
Despite this exception, the process seems OK because it just terminate the
SnapPuller thread but all other threads that process the closing go well.
*Now, the problem is when the close() method called during the reloadCore
doesn't really close the core.*
This is the beginning of the close() method.
public void close() {
int count = refCount.decrementAndGet();
if (count > 0) return; // close is called often, and only actually
closes if nothing is using it.
if (count < 0) {
log.error("Too many close [count:{}] on {}. Please report this
exception to solr-user@lucene.apache.org", count, this );
assert false : "Too many closes on SolrCore";
return;
}
log.info(logid+" CLOSING SolrCore " + this);
When a HTTP Request is executing, the refCount is greater than 1. So, when
the old core is trying to be closed during the core reload, the if (count >
0) condition simply return this method.
Then, fetchLastIndex method in SnapPuller processes next code and execute
"openNewSearcherAndUpdateCommitPoint". If you look at this method, it
tries to open a new searcher of the solrCore which is referenced during the
SnapPuller constructor and I believe this one points to the old core. At
certain timing, this method also throw
SnapPuller - java.lang.InterruptedException
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:680)
After this exception, things start to go bad.
*In summary, I have two questions.*
1. Can you confirm this memory / thread issue?
2. When the core reload happens successfully (no matter it throws the
exception or not), does Solr need to call the
openNewSearcherAndUpdateCommitPoint method?
Thanks.
Re: Memory Issue with SnapPuller
Sorry, wrong button was clicked.
A little more about "At certain timing, this method also throw "
SnapPuller - java.lang.InterruptedException
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:680)
This is a scenario (with less confident).
The old core didn't complete the close() method because of the refCount
during the reloadCore. Then, it execute the
openNewSearcherAndUpdateCommitPoint
method. Now, a http request, as an example, finished a process and called
the SolrCore close() method. refCount is 0. and go to all other process in
the close() method of the SolrCore.
In this case, the InterruptedException can be thrown in the
openNewSearcherAndUpdateCommitPoint. After that, I noticed a one thread
that executes a newSearcher process hangs and high CPU usage remains high.
We are also using a larger external field file too.
On Thu, Oct 20, 2016 at 9:11 AM, Jihwan Kim wrote:
> A little more about "At certain timing, this method also throw "
> SnapPuller - java.lang.InterruptedException
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:680)
>
> This is less confident scenario.
> The old core didn't complete the close() method during the reloadCore.
> Then, it execute the openNewSearcherAndUpdateCommitPoint method. Now, a
> http request, as an example, finished a process and called the SolrCore
> close() method. refCount is 0. and go to all other process in the close()
>
>
> On Thu, Oct 20, 2016 at 8:44 AM, Jihwan Kim wrote:
>
>> Hi,
>> We are using Solr 4.10.4 and experiencing out of memory exception. It
>> seems the problem is cause by the following code & scenario.
>>
>> This is the last part of a fetchLastIndex method in SnapPuller.java
>>
>> // we must reload the core after we open the IW back up
>> if (reloadCore) {
>> reloadCore();
>> }
>>
>> if (successfulInstall) {
>> if (isFullCopyNeeded) {
>> // let the system know we are changing dir's and the old one
>> // may be closed
>> if (indexDir != null) {
>> LOG.info("removing old index directory " + indexDir);
>> core.getDirectoryFactory().doneWithDirectory(indexDir);
>> core.getDirectoryFactory().remove(indexDir);
>> }
>> }
>> if (isFullCopyNeeded) {
>> solrCore.getUpdateHandler().newIndexWriter(isFullCopyNeeded);
>> }
>>
>> openNewSearcherAndUpdateCommitPoint(isFullCopyNeeded);
>> }
>>
>> Inside the reloadCore, it create a new core, register it, and try to
>> close the current/old core. When the closing old core process goes normal,
>> it throws an exception "SnapPull failed
>> :org.apache.solr.common.SolrException:
>> Index fetch failed Caused by java.lang.RuntimeException: Interrupted while
>> waiting for core reload to finish Caused by Caused by:
>> java.lang.InterruptedException."
>>
>> Despite this exception, the process seems OK because it just terminate
>> the SnapPuller thread but all other threads that process the closing go
>> well.
>>
>> *Now, the problem is when the close() method called during the reloadCore
>> doesn't really close the core.*
>> This is the beginning of the close() method.
>> public void close() {
>> int count = refCount.decrementAndGet();
>> if (count > 0) return; // close is called often, and only
>> actually closes if nothing is using it.
>> if (count < 0) {
>>log.error("Too many close [count:{}] on {}. Please report this
>> exception to solr-user@lucene.apache.org", count, this );
>>assert false : "Too many closes on SolrCore";
>>return;
>> }
>> log.info(logid+" CLOSING SolrCore " + this);
>>
>> When a HTTP Request is executing, the refCount is greater than 1. So,
>> when the old core is trying to be closed during the core reload, the if
>> (count > 0) condition simply return this method.
>>
>> Then, fetchLastIndex method in SnapPuller processes next code and execute
>> "openNewSearcherAndUpdateCommitPoint". If you look at this method, it
>> tries to open a new searcher of the solrCore which is referenced duri
Re: Memory Issue with SnapPuller
A little more about "At certain timing, this method also throw "
SnapPuller - java.lang.InterruptedException
at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
at java.util.concurrent.FutureTask.get(FutureTask.java:191)
at SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:680)
This is less confident scenario.
The old core didn't complete the close() method during the reloadCore.
Then, it execute the openNewSearcherAndUpdateCommitPoint method. Now, a
http request, as an example, finished a process and called the SolrCore
close() method. refCount is 0. and go to all other process in the close()
On Thu, Oct 20, 2016 at 8:44 AM, Jihwan Kim wrote:
> Hi,
> We are using Solr 4.10.4 and experiencing out of memory exception. It
> seems the problem is cause by the following code & scenario.
>
> This is the last part of a fetchLastIndex method in SnapPuller.java
>
> // we must reload the core after we open the IW back up
> if (reloadCore) {
> reloadCore();
> }
>
> if (successfulInstall) {
> if (isFullCopyNeeded) {
> // let the system know we are changing dir's and the old one
> // may be closed
> if (indexDir != null) {
> LOG.info("removing old index directory " + indexDir);
> core.getDirectoryFactory().doneWithDirectory(indexDir);
> core.getDirectoryFactory().remove(indexDir);
> }
> }
> if (isFullCopyNeeded) {
> solrCore.getUpdateHandler().newIndexWriter(isFullCopyNeeded);
> }
>
> openNewSearcherAndUpdateCommitPoint(isFullCopyNeeded);
> }
>
> Inside the reloadCore, it create a new core, register it, and try to close
> the current/old core. When the closing old core process goes normal, it
> throws an exception "SnapPull failed :org.apache.solr.common.SolrException:
> Index fetch failed Caused by java.lang.RuntimeException: Interrupted while
> waiting for core reload to finish Caused by Caused by: java.lang.
> InterruptedException."
>
> Despite this exception, the process seems OK because it just terminate the
> SnapPuller thread but all other threads that process the closing go well.
>
> *Now, the problem is when the close() method called during the reloadCore
> doesn't really close the core.*
> This is the beginning of the close() method.
> public void close() {
> int count = refCount.decrementAndGet();
> if (count > 0) return; // close is called often, and only actually
> closes if nothing is using it.
> if (count < 0) {
>log.error("Too many close [count:{}] on {}. Please report this
> exception to solr-user@lucene.apache.org", count, this );
>assert false : "Too many closes on SolrCore";
>return;
> }
> log.info(logid+" CLOSING SolrCore " + this);
>
> When a HTTP Request is executing, the refCount is greater than 1. So, when
> the old core is trying to be closed during the core reload, the if (count >
> 0) condition simply return this method.
>
> Then, fetchLastIndex method in SnapPuller processes next code and execute "
> openNewSearcherAndUpdateCommitPoint". If you look at this method, it
> tries to open a new searcher of the solrCore which is referenced during the
> SnapPuller constructor and I believe this one points to the old core. At
> certain timing, this method also throw
> SnapPuller - java.lang.InterruptedException
> at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
> at java.util.concurrent.FutureTask.get(FutureTask.java:191)
> at SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:680)
>
> After this exception, things start to go bad.
>
> *In summary, I have two questions.*
> 1. Can you confirm this memory / thread issue?
> 2. When the core reload happens successfully (no matter it throws the
> exception or not), does Solr need to call the
> openNewSearcherAndUpdateCommitPoint
> method?
>
> Thanks.
>
Re: Memory Issue with SnapPuller
Thank you Shawn. I understand the two options. After my own testing with a smaller heap, I increased my heap size more than triple, but OOME happens again with my testing cases under the controlled thread process. Increased heap size just delayed the OOME. Can you provide a feedback on my second question: When the core reload happens successfully (no matter it throws the exception or not), does Solr need to call the openNewSearcherAndUpdateCommitPoint method? As I described on my previous email, a thread created from openNewSearcherAndUpdateCommitPoint method hangs and cause a high CPU usage and a slow response time. Attached image is the thread hung. On Thu, Oct 20, 2016 at 9:29 AM, Shawn Heisey wrote: > On 10/20/2016 8:44 AM, Jihwan Kim wrote: > > We are using Solr 4.10.4 and experiencing out of memory exception. It > > seems the problem is cause by the following code & scenario. > > When you get an OutOfMemoryError exception that tells you there's not > enough heap space, the place where the exception happens is frequently > unrelated to the actual source of the problem. Also, unless the > programmer engages in extraordinary effort, encountering OOME will cause > program behavior to become completely unpredictable. Most of Solr has > *NOT* had the benefit of extraordinary effort to handle OOME gracefully. > > Before continuing with troubleshooting of SnapPuller, you're going to > need to fix the OOME error. When you run out of memory, that is likely > to be the CAUSE of any errors you're seeing, not a symptom. > > There are exactly two ways to deal with OOME: Increase the max heap, or > take steps to reduce the amount of heap required. Increasing the heap > is the easiest option, and typically the first step. Sometimes it's the > ONLY option. > > https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > > Thanks, > Shawn > >
Re: Memory Issue with SnapPuller
Good points. I am able to create this with periodic snap puller and only one http request. When I load the Solr on tomcat, the initial memory usage was between 600M to 800 M. First time, I used 1.5 G and then increased the heap to 3.5G. (When I said 'triple', I meant comparing to the initial memory consumption) OK... Shall we focus on my second question: When the core reload happens successfully (no matter it throws the exception or not), does Solr need to call the openNewSearcherAndUpdateCommitPoint method? I think this openNewSearcherAndUpdateCommitPoint method tries to open a new searcher on the old SolrCore. Thank you! On Thu, Oct 20, 2016 at 9:55 AM, Erick Erickson wrote: > You say you tripled the memory. Up to what? Tripling from 500M t0 1.5G > isn't likely enough, tripling from 6G to 18G is something else > again > > You can take a look through any of the memory profilers and try to > catch the objects (and where they're being allocated). The second is > to look at the stack trace (presuming you don't have an OOM killer > script running) and perhaps triangulate that way. > > Best, > Erick > > On Thu, Oct 20, 2016 at 11:44 AM, Jihwan Kim wrote: > > Thank you Shawn. I understand the two options. > > After my own testing with a smaller heap, I increased my heap size more > than > > triple, but OOME happens again with my testing cases under the controlled > > thread process. Increased heap size just delayed the OOME. > > > > Can you provide a feedback on my second question: When the core reload > > happens successfully (no matter it throws the exception or not), does > Solr > > need to call the openNewSearcherAndUpdateCommitPoint method? > > > > As I described on my previous email, a thread created from > > openNewSearcherAndUpdateCommitPoint method hangs and cause a high CPU > usage > > and a slow response time. Attached image is the thread hung. > > > > > > > > On Thu, Oct 20, 2016 at 9:29 AM, Shawn Heisey > wrote: > >> > >> On 10/20/2016 8:44 AM, Jihwan Kim wrote: > >> > We are using Solr 4.10.4 and experiencing out of memory exception. It > >> > seems the problem is cause by the following code & scenario. > >> > >> When you get an OutOfMemoryError exception that tells you there's not > >> enough heap space, the place where the exception happens is frequently > >> unrelated to the actual source of the problem. Also, unless the > >> programmer engages in extraordinary effort, encountering OOME will cause > >> program behavior to become completely unpredictable. Most of Solr has > >> *NOT* had the benefit of extraordinary effort to handle OOME gracefully. > >> > >> Before continuing with troubleshooting of SnapPuller, you're going to > >> need to fix the OOME error. When you run out of memory, that is likely > >> to be the CAUSE of any errors you're seeing, not a symptom. > >> > >> There are exactly two ways to deal with OOME: Increase the max heap, or > >> take steps to reduce the amount of heap required. Increasing the heap > >> is the easiest option, and typically the first step. Sometimes it's the > >> ONLY option. > >> > >> https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > >> > >> Thanks, > >> Shawn > >> > > >
Re: Memory Issue with SnapPuller
This is also the screenshot of jvisualvm. This exception occurred at 2:55PM and 3:40PM and OOME occurs at 3:41PM. SnapPuller - java.lang.InterruptedException at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404) at java.util.concurrent.FutureTask.get(FutureTask.java:191) at SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:680) On Thu, Oct 20, 2016 at 10:14 AM, Jihwan Kim wrote: > Good points. > I am able to create this with periodic snap puller and only one http > request. When I load the Solr on tomcat, the initial memory usage was > between 600M to 800 M. First time, I used 1.5 G and then increased the > heap to 3.5G. (When I said 'triple', I meant comparing to the initial > memory consumption) > > OK... Shall we focus on my second question: When the core reload happens > successfully (no matter it throws the exception or not), does Solr need to > call the openNewSearcherAndUpdateCommitPoint method? I think this > openNewSearcherAndUpdateCommitPoint method tries to open a new searcher > on the old SolrCore. > > Thank you! > > > On Thu, Oct 20, 2016 at 9:55 AM, Erick Erickson > wrote: > >> You say you tripled the memory. Up to what? Tripling from 500M t0 1.5G >> isn't likely enough, tripling from 6G to 18G is something else >> again >> >> You can take a look through any of the memory profilers and try to >> catch the objects (and where they're being allocated). The second is >> to look at the stack trace (presuming you don't have an OOM killer >> script running) and perhaps triangulate that way. >> >> Best, >> Erick >> >> On Thu, Oct 20, 2016 at 11:44 AM, Jihwan Kim wrote: >> > Thank you Shawn. I understand the two options. >> > After my own testing with a smaller heap, I increased my heap size more >> than >> > triple, but OOME happens again with my testing cases under the >> controlled >> > thread process. Increased heap size just delayed the OOME. >> > >> > Can you provide a feedback on my second question: When the core reload >> > happens successfully (no matter it throws the exception or not), does >> Solr >> > need to call the openNewSearcherAndUpdateCommitPoint method? >> > >> > As I described on my previous email, a thread created from >> > openNewSearcherAndUpdateCommitPoint method hangs and cause a high CPU >> usage >> > and a slow response time. Attached image is the thread hung. >> > >> > >> > >> > On Thu, Oct 20, 2016 at 9:29 AM, Shawn Heisey >> wrote: >> >> >> >> On 10/20/2016 8:44 AM, Jihwan Kim wrote: >> >> > We are using Solr 4.10.4 and experiencing out of memory exception. It >> >> > seems the problem is cause by the following code & scenario. >> >> >> >> When you get an OutOfMemoryError exception that tells you there's not >> >> enough heap space, the place where the exception happens is frequently >> >> unrelated to the actual source of the problem. Also, unless the >> >> programmer engages in extraordinary effort, encountering OOME will >> cause >> >> program behavior to become completely unpredictable. Most of Solr has >> >> *NOT* had the benefit of extraordinary effort to handle OOME >> gracefully. >> >> >> >> Before continuing with troubleshooting of SnapPuller, you're going to >> >> need to fix the OOME error. When you run out of memory, that is likely >> >> to be the CAUSE of any errors you're seeing, not a symptom. >> >> >> >> There are exactly two ways to deal with OOME: Increase the max heap, >> or >> >> take steps to reduce the amount of heap required. Increasing the heap >> >> is the easiest option, and typically the first step. Sometimes it's >> the >> >> ONLY option. >> >> >> >> https://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap >> >> >> >> Thanks, >> >> Shawn >> >> >> > >> > >
Re: "on deck" searcher vs warming searcher
why is there a setting (maxWarmingSearchers) that even lets you have more than one: Isn't it also for a case of (frequent) update? For example, one update is committed. During the warming up for this commit, another update is made. In this case the new commit also go through another warming. If the value is 1, the second warming will fail. More number of concurrent warming-up requires larger memory usage. On Fri, Dec 9, 2016 at 9:14 AM, Erick Erickson wrote: > bq: because shouldn't there only be one active > searcher at a time? > > Kind of. This is a total nit, but there can be multiple > searchers serving queries briefly (one hopes at least). > S1 is serving some query when S2 becomes > active and starts getting new queries. Until the last > query S1 is serving is complete, they both are active. > > bq: why is there a setting > (maxWarmingSearchers) that even lets > you have more than one > > The contract is that when you commit (assuming > you're opening a new searcher), then all docs > indexed up to that point are visible. Therefore you > _must_ open a new searcher even if one is currently > warming or that contract would be violated. Since > warming can take minutes, not opening a new > searcher if one was currently warming could cause > quite a gap. > > > Best, > Erick > > On Fri, Dec 9, 2016 at 7:30 AM, Brent wrote: > > Hmmm, conflicting answers. Given the infamous "PERFORMANCE WARNING: > > Overlapping onDeckSearchers" log message, it seems like the "they're the > > same" answer is probably correct, because shouldn't there only be one > active > > searcher at a time? > > > > Although it makes me curious, if there's a warning about having multiple > > (overlapping) warming searchers, why is there a setting > > (maxWarmingSearchers) that even lets you have more than one, or at least, > > why ever set it to anything other than 1? > > > > > > > > -- > > View this message in context: http://lucene.472066.n3. > nabble.com/on-deck-searcher-vs-warming-searcher-tp4309021p4309080.html > > Sent from the Solr - User mailing list archive at Nabble.com. >
