I tried raising the <maxFieldLength>100000000</maxFieldLength> under <mainIndex> as well as <indexDefaults> and still no luck. I'm trying to upload a text file that is about 8 MB in size. I think the following stack trace still points to some sort of overflowed String issue. Thoughts? Solr returned an error: Java heap space java.lang.OutOfMemoryError: Java heap space at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232) at java.lang.StringCoding.encode(StringCoding.java:272) at java.lang.String.getBytes(String.java:947) at org.apache.lucene.index.FieldsWriter.addDocument(FieldsWriter.java:98) at org.apache.lucene.index.DocumentWriter.addDocument(DocumentWriter.java:107)
at org.apache.lucene.index.IndexWriter.buildSingleDocSegment( IndexWriter.java:977) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:965) at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:947) at org.apache.solr.update.DirectUpdateHandler2.addDoc( DirectUpdateHandler2.java:270) at org.apache.solr.handler.XmlUpdateRequestHandler.update( XmlUpdateRequestHandler.java:166) at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody( XmlUpdateRequestHandler.java:84) at org.apache.solr.handler.RequestHandlerBase.handleRequest( RequestHandlerBase.java:77) at org.apache.solr.core.SolrCore.execute(SolrCore.java:658) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:191) at org.apache.solr.servlet.SolrDispatchFilter.doFilter( SolrDispatchFilter.java:159) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter( ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke( StandardWrapperValve.java:213) at org.apache.catalina.core.StandardContextValve.invoke( StandardContextValve.java:174) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java :127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java :117) at org.apache.catalina.core.StandardEngineValve.invoke( StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection (Http11BaseProtocol.java:665) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket( PoolTcpEndpoint.java:528) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt( LeaderFollowerWorkerThread.java:81) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run( ThreadPool.java:689) at java.lang.Thread.run(Thread.java:619) java.io.IOException: Server returned HTTP response code: 500 for URL: http://solr:8080/solr/update at sun.net.www.protocol.http.HttpURLConnection.getInputStream( HttpURLConnection.java:1170) at com.itstrategypartners.sents.solrUpload.SimplePostTool.postData( SimplePostTool.java:134) at com.itstrategypartners.sents.solrUpload.SimplePostTool.postFile( SimplePostTool.java:87) at com.itstrategypartners.sents.solrUpload.Uploader.uploadFile( Uploader.java:97) at com.itstrategypartners.sents.solrUpload.UploaderTest.uploadFile( UploaderTest.java:95) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke( NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke( DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.junit.internal.runners.TestMethodRunner.executeMethodBody( TestMethodRunner.java:99) at org.junit.internal.runners.TestMethodRunner.runUnprotected( TestMethodRunner.java:81) at org.junit.internal.runners.BeforeAndAfterRunner.runProtected( BeforeAndAfterRunner.java:34) at org.junit.internal.runners.TestMethodRunner.runMethod( TestMethodRunner.java:75) at org.junit.internal.runners.TestMethodRunner.run( TestMethodRunner.java:45) at org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod( TestClassMethodsRunner.java:71) at org.junit.internal.runners.TestClassMethodsRunner.run( TestClassMethodsRunner.java:35) at org.junit.internal.runners.TestClassRunner$1.runUnprotected( TestClassRunner.java:42) at org.junit.internal.runners.BeforeAndAfterRunner.runProtected( BeforeAndAfterRunner.java:34) at org.junit.internal.runners.TestClassRunner.run( TestClassRunner.java:52) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:32) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run( JUnitTestRunner.java:421) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch( JUnitTestRunner.java:912) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main (JUnitTestRunner.java:766) On 1/16/08, David Thibault <[EMAIL PROTECTED]> wrote: > > I think your PS might do the trick. My JVM doesn't seem to be the issue, > because I've set it to -Xmx512m -Xms256m. I will track down the solr config > parameter you mentioned and try that. Thanks for the quick response! > > Dave > > On 1/16/08, Erick Erickson <[EMAIL PROTECTED] > wrote: > > > > P.S. Lucene by default limits the maximum field length > > to 10K tokens, so you have to bump that for large files. > > > > Erick > > > > On Jan 16, 2008 11:04 AM, Erick Erickson <[EMAIL PROTECTED]> > > wrote: > > > > > I don't think this is a StringBuilder limitation, but rather your Java > > > > > JVM doesn't start with enough memory. i.e. -Xmx. > > > > > > In raw Lucene, I've indexed 240M files........ > > > > > > Best > > > Erick > > > > > > > > > On Jan 16, 2008 10:12 AM, David Thibault < [EMAIL PROTECTED] > > > > > > wrote: > > > > > > > All, > > > > I just found a thread about this on the mailing list archives > > because > > > > I'm > > > > troubleshooting the same problem. The kicker is that it doesn't > > take > > > > such > > > > large files to kill the StringBuilder. I have discovered the > > following: > > > > > > > > > > > > By using a text file made up of 3,443,464 bytes or less, I get no > > > > error. > > > > > > > > AT 3,443,465 bytes: > > > > > > > > > > > > Exception in thread "main" java.lang.OutOfMemoryError : Java heap > > space > > > > > > > > at java.lang.String .<init>(String.java:208) > > > > > > > > at java.lang.StringBuilder.toString(StringBuilder.java:431) > > > > > > > > at org.junit.Assert.format(Assert.java:321) > > > > > > > > at org.junit.ComparisonFailure$ComparisonCompactor.compact ( > > > > ComparisonFailure.java:80) > > > > > > > > at org.junit.ComparisonFailure.getMessage ( > > ComparisonFailure.java > > > > :37) > > > > > > > > at java.lang.Throwable.getLocalizedMessage(Throwable.java > > :267) > > > > > > > > at java.lang.Throwable.toString (Throwable.java:344) > > > > > > > > at java.lang.String.valueOf(String.java:2615) > > > > > > > > at java.io.PrintWriter.print(PrintWriter.java:546) > > > > > > > > at java.io.PrintWriter.println (PrintWriter.java:683) > > > > > > > > at java.lang.Throwable.printStackTrace(Throwable.java:510) > > > > > > > > at org.apache.tools.ant.util.StringUtils.getStackTrace( > > > > StringUtils.java :96) > > > > > > > > at > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.getFilteredTrace > > > > (JUnitTestRunner.java:856) > > > > > > > > at > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.formatError > > > > (XMLJUnitResultFormatter.java:280) > > > > > > > > at > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.addError > > > > (XMLJUnitResultFormatter.java:255) > > > > > > > > at > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner$4.addError( > > > > JUnitTestRunner.java:988) > > > > > > > > at junit.framework.TestResult.addError(TestResult.java :38) > > > > > > > > at junit.framework.JUnit4TestAdapterCache$1.testFailure( > > > > JUnit4TestAdapterCache.java:51) > > > > > > > > at org.junit.runner.notification.RunNotifier$4.notifyListener > > ( > > > > RunNotifier.java:96) > > > > > > > > at org.junit.runner.notification.RunNotifier$SafeNotifier.run > > ( > > > > RunNotifier.java:37) > > > > > > > > at org.junit.runner.notification.RunNotifier.fireTestFailure( > > > > RunNotifier.java:93) > > > > > > > > at org.junit.internal.runners.TestMethodRunner.addFailure ( > > > > TestMethodRunner.java:104) > > > > > > > > at org.junit.internal.runners.TestMethodRunner.runUnprotected > > ( > > > > TestMethodRunner.java:87) > > > > > > > > at > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected ( > > > > BeforeAndAfterRunner.java:34) > > > > > > > > at org.junit.internal.runners.TestMethodRunner.runMethod( > > > > TestMethodRunner.java:75) > > > > > > > > at org.junit.internal.runners.TestMethodRunner.run ( > > > > TestMethodRunner.java :45) > > > > > > > > at > > > > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod( > > > > TestClassMethodsRunner.java:71) > > > > > > > > at org.junit.internal.runners.TestClassMethodsRunner.run( > > > > TestClassMethodsRunner.java :35) > > > > > > > > at > > org.junit.internal.runners.TestClassRunner$1.runUnprotected( > > > > TestClassRunner.java :42) > > > > > > > > at > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > > > > BeforeAndAfterRunner.java:34) > > > > > > > > at org.junit.internal.runners.TestClassRunner.run ( > > > > TestClassRunner.java:52) > > > > > > > > at junit.framework.JUnit4TestAdapter.run( > > JUnit4TestAdapter.java > > > > :32) > > > > > > > > > > > > > > > > AT 3,443,466 byes (or more) : > > > > > > > > > > > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap > > space > > > > > > > > at java.lang.AbstractStringBuilder.expandCapacity( > > > > AbstractStringBuilder.java :99) > > > > > > > > at java.lang.AbstractStringBuilder.append ( > > > > AbstractStringBuilder.java > > > > :393) > > > > > > > > at java.lang.StringBuilder.append(StringBuilder.java :120) > > > > > > > > at org.junit.Assert.format(Assert.java:321) > > > > > > > > at org.junit.ComparisonFailure$ComparisonCompactor.compact ( > > > > ComparisonFailure.java:80) > > > > > > > > at org.junit.ComparisonFailure.getMessage( > > ComparisonFailure.java > > > > :37) > > > > > > > > at java.lang.Throwable.getLocalizedMessage(Throwable.java > > :267) > > > > > > > > at java.lang.Throwable.toString (Throwable.java:344) > > > > > > > > at java.lang.String.valueOf(String.java:2615) > > > > > > > > at java.io.PrintWriter.print(PrintWriter.java:546) > > > > > > > > at java.io.PrintWriter.println(PrintWriter.java:683) > > > > > > > > at java.lang.Throwable.printStackTrace(Throwable.java:510) > > > > > > > > at org.apache.tools.ant.util.StringUtils.getStackTrace ( > > > > StringUtils.java:96) > > > > > > > > at > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.getFilteredTrace > > > > (JUnitTestRunner.java:856) > > > > > > > > at > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.formatError > > > > (XMLJUnitResultFormatter.java:280) > > > > > > > > at > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.addError > > > > (XMLJUnitResultFormatter.java:255) > > > > > > > > at > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner$4.addError( > > > > JUnitTestRunner.java:988) > > > > > > > > at junit.framework.TestResult.addError(TestResult.java :38) > > > > > > > > at junit.framework.JUnit4TestAdapterCache$1.testFailure ( > > > > JUnit4TestAdapterCache.java:51) > > > > > > > > at org.junit.runner.notification.RunNotifier$4.notifyListener > > ( > > > > RunNotifier.java:96) > > > > > > > > at org.junit.runner.notification.RunNotifier$SafeNotifier.run( > > > > RunNotifier.java:37) > > > > > > > > at org.junit.runner.notification.RunNotifier.fireTestFailure( > > > > RunNotifier.java:93) > > > > > > > > at org.junit.internal.runners.TestMethodRunner.addFailure ( > > > > TestMethodRunner.java:104) > > > > > > > > at org.junit.internal.runners.TestMethodRunner.runUnprotected > > ( > > > > TestMethodRunner.java:87) > > > > > > > > at > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected ( > > > > BeforeAndAfterRunner.java:34) > > > > > > > > at org.junit.internal.runners.TestMethodRunner.runMethod( > > > > TestMethodRunner.java:75) > > > > > > > > at org.junit.internal.runners.TestMethodRunner.run ( > > > > TestMethodRunner.java :45) > > > > > > > > at > > > > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod( > > > > TestClassMethodsRunner.java:71) > > > > > > > > at org.junit.internal.runners.TestClassMethodsRunner.run( > > > > TestClassMethodsRunner.java :35) > > > > > > > > at > > org.junit.internal.runners.TestClassRunner$1.runUnprotected( > > > > TestClassRunner.java :42) > > > > > > > > at > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > > > > BeforeAndAfterRunner.java:34) > > > > > > > > at org.junit.internal.runners.TestClassRunner.run ( > > > > TestClassRunner.java:52) > > > > > > > > > > > > I am writing a filesystem crawler so I need to be able to crawl and > > > > index > > > > any size file (within reason). A 3-4MB file is certainly within > > reason. > > > > I > > > > rewrote my code to store the file contents in a file and read/write > > in > > > > one > > > > line at a time. However, when I post the XML file to Solr using > > > > SimplePostTool, I get another OutOfMemoryError about the java heap > > space > > > > > > > > (thrown from org.xmlpull... again). In any case, does anyone have > > any > > > > ideas > > > > about this? Has anyone posted documents with contents larger than > > 3.5MBto > > > > Solr successfully? If so, how was it done? I'm using Solr v1.2. > > > > > > > > > > > > Best, > > > > > > > > Dave > > > > > > > > > > > > > > >
