I don't really follow DataImportHandler, but it looks like its using an
unbounded cache (simple HashMap).
Perhaps we should make the cache size configurable?
The impl seems a little odd - the caching occurs in the base class - so
caching impls that extends it don't really have full control - they just
kind of "turn on" the caching in the base class? Kind of an odd approach
- to cache you have to turn on the cache support in the base class and
impl a couple custom methods as well?
--
- Mark
http://www.lucidimagination.com
On 03/13/2010 06:11 PM, JavaGuy84 wrote:
Erik,
I have seen many posts regarding out of memory error but I am not sure
whether they are using cachesqlEntityProcessor..
I want to know if there is a way to flush out the buffer of cache instead of
storing everything in cache.
I can clearly see the heapsize growing like anything if I use the
cachesqlentity processor, trying to figure out if there is a way to resolve
this by using any other way other than using this processor.
Thanks,
Barani
Erick Erickson wrote:
Have you searched the users' list? This question has come up multiple
times
and you'll find your question has probably already been answered. Let us
know if you come up blank...
Best
Erick
On Sat, Mar 13, 2010 at 3:56 PM, JavaGuy84<bbar...@gmail.com> wrote:
Sorry forgot to attach the error log,
Error Log:
---------
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMe
moryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:650)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:605)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j
ava:261)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java
:185)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo
rter.java:333)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j
ava:391)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja
va:372)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.<init>(Unknown Source)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.g
etARow(JdbcDataSource.java:281)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.a
ccess$800(JdbcDataSource.java:228)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:266)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:269)
at
org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.getAllNon
CachedRows(CachedSqlEntityProcessor.java:70)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getIdCacheData
(EntityProcessorBase.java:194)
at
org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.nextRow(C
achedSqlEntityProcessor.java:58)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(Ent
ityProcessorWrapper.java:233)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:579)
... 6 more
Mar 13, 2010 3:52:09 PM org.apache.solr.handler.dataimport.DataImporter
doFullIm
port
SEVERE: Full Import failed
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMe
moryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:650)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:605)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j
ava:261)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java
:185)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo
rter.java:333)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j
ava:391)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja
va:372)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.<init>(Unknown Source)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.g
etARow(JdbcDataSource.java:281)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.a
ccess$800(JdbcDataSource.java:228)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:266)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:269)
at
org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.getAllNon
CachedRows(CachedSqlEntityProcessor.java:70)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getIdCacheData
(EntityProcessorBase.java:194)
at
org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.nextRow(C
achedSqlEntityProcessor.java:58)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(Ent
ityProcessorWrapper.java:233)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:579)
... 6 more
Mar 13, 2010 3:52:09 PM org.apache.solr.update.DirectUpdateHandler2
rollback
INFO: start rollback
Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.<init>(Unknown Source)
at java.util.ArrayList.<init>(Unknown Source)
at org.mortbay.util.Scanner.reportDifferences(Scanner.java:289)
at org.mortbay.util.Scanner.scan(Scanner.java:254)
at org.mortbay.util.Scanner$1.run(Scanner.java:225)
at java.util.TimerThread.mainLoop(Unknown Source)
Exception in thread "SocketServerInstance-crprdnii1soe-31000" at
java.util.Tim
erThread.run(Unknown Source)
java.lang.OutOfMemoryError: Java heap space
at java.nio.CharBuffer.wrap(Unknown Source)
at sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
at sun.nio.cs.StreamEncoder.write(Unknown Source)
at java.io.OutputStreamWriter.write(Unknown Source)
at java.io.BufferedWriter.flushBuffer(Unknown Source)
at java.io.PrintStream.write(Unknown Source)
at java.io.PrintStream.print(Unknown Source)
at
com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.log(P
rintStreamSocketLog.java:169)
at
com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.log(P
rintStreamSocketLog.java:175)
at
com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.logEr
ror(PrintStreamSocketLog.java:71)
at
com.metamatrix.common.comm.platform.socket.client.SocketServerInstanc
eImpl$1.run(SocketServerInstanceImpl.java:578)
at java.lang.Thread.run(Unknown Source)
Mar 13, 2010 3:52:09 PM org.apache.solr.update.DirectUpdateHandler2
rollback
INFO: end_rollback
Thanks,
Barani
JavaGuy84 wrote:
Hi,
I am using CachedsqlEntityProcessor in my DIH dataconfig to reduce the
number of queries executed against the database ,
<Entity1 query="select * from x" processor=
"CachedsqlEntityProcessor"/>
<Entity2 query="select * from y" processor= "CachedsqlEntityProcessor"
cachekey="id" cachelookup="x.id"/>
I having more than 2 million rows returned for Entity 2 and around
300000
rows returned for entity1.
I am have set the heap size to 1 GB but even then I am always getting
heap
out of size error. I am not sure how to flush the documents in buffer
at
certain condition. I tried to enable Autocommit / reduced the
maxdocbuffersize but of no use.. Can someone let me know what is the
best
way to overcome this issue?
Thanks,
Barani
--
View this message in context:
http://old.nabble.com/DIH---Out-of-Memory-error-when-using-CachedsqlEntityProcessor-tp27889623p27890751.html
Sent from the Solr - User mailing list archive at Nabble.com.
