Find attached the relevant part of the batch-update:
...
SolrClient solrClient = getSolrClient( coreName, true );
Collection<SolrInputDocument> batch = new ArrayList<SolrInputDocument>();
while ( elements.hasNext() )
{
IIndexableElement elem = elements.next();
SolrInputDocument doc = createSolrDocForElement( elem, provider, locale ); //
[1]
if ( doc != null )
{
batch.add( doc );
if ( batch.size() == 100 )
{
solrClient.add( documents ); // [2]
batch = new ArrayList<SolrInputDocument>(); // [3]
}
}
}
if ( !batch.isEmpty() )
{
addMultipleDocuments( uniqueProviderName, solrClient, batch );
batch = null;
}
...
IIndexableElement is part of our index/search framework.
[1] creating a single SolrInputDocument
[2] handing the SIDs to SolrJ/SolrClient
[3] creating a new batch, i.e. releasing the SolrInputDocuments
The above code is being executed in an ExecutorService handed in as a lambda.
I.e.:
executorService.submit( () -> {
<the code>
} );
Thanks for any advices. If needed, I can also provide the OOM-heapdump ...
-----Ursprüngliche Nachricht-----
Von: Shawn Heisey [mailto:apa...@elyograg.org]
Gesendet: Freitag, 19. Februar 2016 18:59
An: solr-user@lucene.apache.org
Betreff: Re: AW: OutOfMemory when batchupdating from SolrJ
On 2/19/2016 3:08 AM, Clemens Wyss DEV wrote:
> The logic is somewhat this:
>
> SolrClient solrClient = new HttpSolrClient( coreUrl ); while ( got
> more elements to index ) {
> batch = create 100 SolrInputDocuments
> solrClient.add( batch )
> }
How much data is going into each of those SolrInputDocument objects?
If the amount of data is very small (a few kilobytes), then this sounds like
your program has a memory leak. Can you provide more code detail?
Ideally, you would make the entire code available by placing it on the Internet
somewhere and providing a URL. If there's anything sensitive in the code, like
passwords or public IP addresses, feel free to redact it, but try not to remove
anything that affects how the code operates.
Thanks,
Shawn
