On 5/22/2014 8:31 AM, Aniket Bhoi wrote:
> On Thu, May 22, 2014 at 7:13 PM, Shawn Heisey <s...@elyograg.org> wrote:
>
>> On 5/22/2014 1:53 AM, Aniket Bhoi wrote:
>>> Details:
>>>
>>> *Solr Version:*
>>> Solr Specification Version: 3.4.0.2012.01.23.14.08.01
>>> Solr Implementation Version: 3.4
>>> Lucene Specification Version: 3.4
>>> Lucene Implementation Version: 3.4
>>>
>>> *Tomcat version:*
>>> Apache Tomcat/6.0.18
>>>
>>> *OS details:*
>>> SUSE Linux Enterprise Server 11 (x86_64)
>>> VERSION = 11
>>> PATCHLEVEL = 1
>>>
>>> While running indexing on this server,It failed.
>>>
>>> Log excerpt:
>>>
>>> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
>>> com.microsoft.sqlserver.jdbc.SQLServerException: The result set is
>> closed.
>>>
>>> Out intial hypothesis was that there is a problem with the connection
>>> thread,so we made changes to the context.xml and added
>>> validationQuery,testOnBorrow etc..to make sure the thread doesnt time
>> out.
>>> We also killed a lot of sleeping sessions from the server to the
>> database.
>>> All of the above still didnt work
>> I have reduced your log excerpt to what I think is the important part.
>>
>> Removing the multithreaded support as others have suggested is a good
>> idea, but what I think is really happening here is that Solr is engaging
>> in a multi-tier merge, so it stops indexing for a while ... and
>> meanwhile, JDBC times out and closes your database connection because of
>> inactivity. When the largest merge tier finishes, indexing tries to
>> resume, which it can't do because the database connection is closed.
>>
>> The solution is to allow more simultaneous merges to happen, which
>> allows indexing to continue while a multi-tier merge is underway. This
>> is my indexConfig section from solrconfig.xml:
>>
>> <indexConfig>
>> <mergePolicy class="org.apache.lucene.index.TieredMergePolicy">
>> <int name="maxMergeAtOnce">35</int>
>> <int name="segmentsPerTier">35</int>
>> <int name="maxMergeAtOnceExplicit">105</int>
>> </mergePolicy>
>> <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
>> <int name="maxThreadCount">1</int>
>> <int name="maxMergeCount">6</int>
>> </mergeScheduler>
>> <ramBufferSizeMB>48</ramBufferSizeMB>
>> <infoStream file="INFOSTREAM-${solr.core.name}.txt">false</infoStream>
>> </indexConfig>
>>
>> The important part for your purposes is the mergeScheduler config, and
>> in particular, maxMergeCount. Increase that to 6. If you are using
>> standard spinning hard disks, do not increase maxThreadCount beyond 1.
>> If you are using SSD, you can safely increase that a small amount, but I
>> don't think I'd go above 2 or 3.
>>
>> Thanks,
>> Shawn
>>
>>
> I may be missing something ,or looking in the wrong place,But I cannot find
> an indexConfig section or any other mentioned detail above in the
> solrconfig.xml file
Solr will work without one, in which case it will simply use the
defaults. With older 3.x versions the mergeScheduler config will
actually need to go in an indexDefaults section. The mainIndex and
indexDefaults sections were deprecated in 3.6 and removed entirely in 4.x.
https://issues.apache.org/jira/browse/SOLR-1052
If you don't have indexDefaults either, you may need to add the config
as a top level element under <config>. If you do this, here's what you
should add:
<indexConfig>
<mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">
<int name="maxThreadCount">1</int>
<int name="maxMergeCount">6</int>
</mergeScheduler>
</indexConfig>
I think we should probably change the default value that Solr uses for
maxMergeCount. This problem comes up fairly often. As long as
maxThreadCount is 1, I cannot think of a really good reason to limit
maxMergeCount at the level that we currently do.
Thanks,
Shawn
