Sorry forgot to attach the error log,
Error Log:
---------
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMe
moryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:650)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:605)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j
ava:261)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java
:185)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo
rter.java:333)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j
ava:391)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja
va:372)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.<init>(Unknown Source)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.g
etARow(JdbcDataSource.java:281)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.a
ccess$800(JdbcDataSource.java:228)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:266)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:269)
at
org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.getAllNon
CachedRows(CachedSqlEntityProcessor.java:70)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getIdCacheData
(EntityProcessorBase.java:194)
at
org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.nextRow(C
achedSqlEntityProcessor.java:58)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(Ent
ityProcessorWrapper.java:233)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:579)
... 6 more
Mar 13, 2010 3:52:09 PM org.apache.solr.handler.dataimport.DataImporter
doFullIm
port
SEVERE: Full Import failed
org.apache.solr.handler.dataimport.DataImportHandlerException:
java.lang.OutOfMe
moryError: Java heap space
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:650)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:605)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.j
ava:261)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java
:185)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImpo
rter.java:333)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.j
ava:391)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.ja
va:372)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.<init>(Unknown Source)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.g
etARow(JdbcDataSource.java:281)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.a
ccess$800(JdbcDataSource.java:228)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:266)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1
.next(JdbcDataSource.java:269)
at
org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.getAllNon
CachedRows(CachedSqlEntityProcessor.java:70)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getIdCacheData
(EntityProcessorBase.java:194)
at
org.apache.solr.handler.dataimport.CachedSqlEntityProcessor.nextRow(C
achedSqlEntityProcessor.java:58)
at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(Ent
ityProcessorWrapper.java:233)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilde
r.java:579)
... 6 more
Mar 13, 2010 3:52:09 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: start rollback
Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space
at java.util.ArrayList.<init>(Unknown Source)
at java.util.ArrayList.<init>(Unknown Source)
at org.mortbay.util.Scanner.reportDifferences(Scanner.java:289)
at org.mortbay.util.Scanner.scan(Scanner.java:254)
at org.mortbay.util.Scanner$1.run(Scanner.java:225)
at java.util.TimerThread.mainLoop(Unknown Source)
Exception in thread "SocketServerInstance-crprdnii1soe-31000" at
java.util.Tim
erThread.run(Unknown Source)
java.lang.OutOfMemoryError: Java heap space
at java.nio.CharBuffer.wrap(Unknown Source)
at sun.nio.cs.StreamEncoder.implWrite(Unknown Source)
at sun.nio.cs.StreamEncoder.write(Unknown Source)
at java.io.OutputStreamWriter.write(Unknown Source)
at java.io.BufferedWriter.flushBuffer(Unknown Source)
at java.io.PrintStream.write(Unknown Source)
at java.io.PrintStream.print(Unknown Source)
at
com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.log(P
rintStreamSocketLog.java:169)
at
com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.log(P
rintStreamSocketLog.java:175)
at
com.metamatrix.common.comm.platform.socket.PrintStreamSocketLog.logEr
ror(PrintStreamSocketLog.java:71)
at
com.metamatrix.common.comm.platform.socket.client.SocketServerInstanc
eImpl$1.run(SocketServerInstanceImpl.java:578)
at java.lang.Thread.run(Unknown Source)
Mar 13, 2010 3:52:09 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback
Thanks,
Barani
JavaGuy84 wrote:
>
> Hi,
>
> I am using CachedsqlEntityProcessor in my DIH dataconfig to reduce the
> number of queries executed against the database ,
>
> <Entity1 query="select * from x" processor= "CachedsqlEntityProcessor"/>
> <Entity2 query="select * from y" processor= "CachedsqlEntityProcessor"
> cachekey="id" cachelookup="x.id"/>
>
>
> I having more than 2 million rows returned for Entity 2 and around 300000
> rows returned for entity1.
>
> I am have set the heap size to 1 GB but even then I am always getting heap
> out of size error. I am not sure how to flush the documents in buffer at
> certain condition. I tried to enable Autocommit / reduced the
> maxdocbuffersize but of no use.. Can someone let me know what is the best
> way to overcome this issue?
>
> Thanks,
> Barani
>
--
View this message in context:
http://old.nabble.com/DIH---Out-of-Memory-error-when-using-CachedsqlEntityProcessor-tp27889623p27890751.html
Sent from the Solr - User mailing list archive at Nabble.com.
