The PS really wasn't related to your OOM, and raising that shouldn't have changed the behavior. All that happens if you go beyond 10,000 tokens is that the rest gets thrown away.
But we're beyond my real knowledge level about SOLR, so I'll defer to others. A very quick-n-dirty test as to whether you're actually allocation more memory to the process you *think* you are would be to bump it ridiculously higher. I'm completely unclear about what process gets the increased memory relative to the server. [EMAIL PROTECTED] On Jan 16, 2008 11:33 AM, David Thibault <[EMAIL PROTECTED]> wrote: > I tried raising the <maxFieldLength>100000000</maxFieldLength> under > <mainIndex> as well as <indexDefaults> and still no luck. I'm trying to > upload a text file that is about 8 MB in size. I think the following > stack > trace still points to some sort of overflowed String issue. Thoughts? > Solr returned an error: Java heap space java.lang.OutOfMemoryError: Java > heap space > at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232) > at java.lang.StringCoding.encode(StringCoding.java:272) > at java.lang.String.getBytes(String.java:947) > at org.apache.lucene.index.FieldsWriter.addDocument(FieldsWriter.java:98) > at org.apache.lucene.index.DocumentWriter.addDocument(DocumentWriter.java > :107) > > at org.apache.lucene.index.IndexWriter.buildSingleDocSegment( > IndexWriter.java:977) > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:965) > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:947) > at org.apache.solr.update.DirectUpdateHandler2.addDoc( > DirectUpdateHandler2.java:270) > at org.apache.solr.handler.XmlUpdateRequestHandler.update( > XmlUpdateRequestHandler.java:166) > at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody( > XmlUpdateRequestHandler.java:84) > at org.apache.solr.handler.RequestHandlerBase.handleRequest( > RequestHandlerBase.java:77) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:658) at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java > :191) > > at org.apache.solr.servlet.SolrDispatchFilter.doFilter( > SolrDispatchFilter.java:159) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( > ApplicationFilterChain.java:215) > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > ApplicationFilterChain.java:188) > at org.apache.catalina.core.StandardWrapperValve.invoke( > StandardWrapperValve.java:213) > at org.apache.catalina.core.StandardContextValve.invoke( > StandardContextValve.java:174) > at org.apache.catalina.core.StandardHostValve.invoke( > StandardHostValve.java > :127) > at org.apache.catalina.valves.ErrorReportValve.invoke( > ErrorReportValve.java > :117) > at org.apache.catalina.core.StandardEngineValve.invoke( > StandardEngineValve.java:108) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java > :151) > > at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java > :874) > > at > > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection > (Http11BaseProtocol.java:665) > at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket( > PoolTcpEndpoint.java:528) > at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt( > LeaderFollowerWorkerThread.java:81) > at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run( > ThreadPool.java:689) > at java.lang.Thread.run(Thread.java:619) > > java.io.IOException: Server returned HTTP response code: 500 for URL: > http://solr:8080/solr/update > at sun.net.www.protocol.http.HttpURLConnection.getInputStream( > HttpURLConnection.java:1170) > at com.itstrategypartners.sents.solrUpload.SimplePostTool.postData( > SimplePostTool.java:134) > at com.itstrategypartners.sents.solrUpload.SimplePostTool.postFile( > SimplePostTool.java:87) > at com.itstrategypartners.sents.solrUpload.Uploader.uploadFile( > Uploader.java:97) > at com.itstrategypartners.sents.solrUpload.UploaderTest.uploadFile( > UploaderTest.java:95) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:39) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at org.junit.internal.runners.TestMethodRunner.executeMethodBody( > TestMethodRunner.java:99) > at org.junit.internal.runners.TestMethodRunner.runUnprotected( > TestMethodRunner.java:81) > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > BeforeAndAfterRunner.java:34) > at org.junit.internal.runners.TestMethodRunner.runMethod( > TestMethodRunner.java:75) > at org.junit.internal.runners.TestMethodRunner.run( > TestMethodRunner.java:45) > at > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod( > TestClassMethodsRunner.java:71) > at org.junit.internal.runners.TestClassMethodsRunner.run( > TestClassMethodsRunner.java:35) > at org.junit.internal.runners.TestClassRunner$1.runUnprotected( > TestClassRunner.java:42) > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > BeforeAndAfterRunner.java:34) > at org.junit.internal.runners.TestClassRunner.run( > TestClassRunner.java:52) > at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:32) > at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run > ( > JUnitTestRunner.java:421) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch( > JUnitTestRunner.java:912) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main > (JUnitTestRunner.java:766) > > On 1/16/08, David Thibault <[EMAIL PROTECTED]> wrote: > > > > I think your PS might do the trick. My JVM doesn't seem to be the > issue, > > because I've set it to -Xmx512m -Xms256m. I will track down the solr > config > > parameter you mentioned and try that. Thanks for the quick response! > > > > Dave > > > > On 1/16/08, Erick Erickson <[EMAIL PROTECTED] > wrote: > > > > > > P.S. Lucene by default limits the maximum field length > > > to 10K tokens, so you have to bump that for large files. > > > > > > Erick > > > > > > On Jan 16, 2008 11:04 AM, Erick Erickson <[EMAIL PROTECTED]> > > > wrote: > > > > > > > I don't think this is a StringBuilder limitation, but rather your > Java > > > > > > > JVM doesn't start with enough memory. i.e. -Xmx. > > > > > > > > In raw Lucene, I've indexed 240M files........ > > > > > > > > Best > > > > Erick > > > > > > > > > > > > On Jan 16, 2008 10:12 AM, David Thibault < > [EMAIL PROTECTED] > > > > > > > > wrote: > > > > > > > > > All, > > > > > I just found a thread about this on the mailing list archives > > > because > > > > > I'm > > > > > troubleshooting the same problem. The kicker is that it doesn't > > > take > > > > > such > > > > > large files to kill the StringBuilder. I have discovered the > > > following: > > > > > > > > > > > > > > > By using a text file made up of 3,443,464 bytes or less, I get no > > > > > error. > > > > > > > > > > AT 3,443,465 bytes: > > > > > > > > > > > > > > > Exception in thread "main" java.lang.OutOfMemoryError : Java heap > > > space > > > > > > > > > > at java.lang.String .<init>(String.java:208) > > > > > > > > > > at java.lang.StringBuilder.toString(StringBuilder.java:431) > > > > > > > > > > at org.junit.Assert.format(Assert.java:321) > > > > > > > > > > at org.junit.ComparisonFailure$ComparisonCompactor.compact( > > > > > ComparisonFailure.java:80) > > > > > > > > > > at org.junit.ComparisonFailure.getMessage ( > > > ComparisonFailure.java > > > > > :37) > > > > > > > > > > at java.lang.Throwable.getLocalizedMessage(Throwable.java > > > :267) > > > > > > > > > > at java.lang.Throwable.toString (Throwable.java:344) > > > > > > > > > > at java.lang.String.valueOf(String.java:2615) > > > > > > > > > > at java.io.PrintWriter.print(PrintWriter.java:546) > > > > > > > > > > at java.io.PrintWriter.println (PrintWriter.java:683) > > > > > > > > > > at java.lang.Throwable.printStackTrace(Throwable.java:510) > > > > > > > > > > at org.apache.tools.ant.util.StringUtils.getStackTrace( > > > > > StringUtils.java :96) > > > > > > > > > > at > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.getFilteredTrace > > > > > (JUnitTestRunner.java:856) > > > > > > > > > > at > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.formatError > > > > > (XMLJUnitResultFormatter.java:280) > > > > > > > > > > at > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.addError > > > > > (XMLJUnitResultFormatter.java:255) > > > > > > > > > > at > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner$4.addError( > > > > > JUnitTestRunner.java:988) > > > > > > > > > > at junit.framework.TestResult.addError(TestResult.java :38) > > > > > > > > > > at junit.framework.JUnit4TestAdapterCache$1.testFailure( > > > > > JUnit4TestAdapterCache.java:51) > > > > > > > > > > at > org.junit.runner.notification.RunNotifier$4.notifyListener > > > ( > > > > > RunNotifier.java:96) > > > > > > > > > > at > org.junit.runner.notification.RunNotifier$SafeNotifier.run > > > ( > > > > > RunNotifier.java:37) > > > > > > > > > > at > org.junit.runner.notification.RunNotifier.fireTestFailure( > > > > > RunNotifier.java:93) > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.addFailure ( > > > > > TestMethodRunner.java:104) > > > > > > > > > > at > org.junit.internal.runners.TestMethodRunner.runUnprotected > > > ( > > > > > TestMethodRunner.java:87) > > > > > > > > > > at > > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected ( > > > > > BeforeAndAfterRunner.java:34) > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.runMethod( > > > > > TestMethodRunner.java:75) > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.run ( > > > > > TestMethodRunner.java :45) > > > > > > > > > > at > > > > > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod > ( > > > > > TestClassMethodsRunner.java:71) > > > > > > > > > > at org.junit.internal.runners.TestClassMethodsRunner.run( > > > > > TestClassMethodsRunner.java :35) > > > > > > > > > > at > > > org.junit.internal.runners.TestClassRunner$1.runUnprotected( > > > > > TestClassRunner.java :42) > > > > > > > > > > at > > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > > > > > BeforeAndAfterRunner.java:34) > > > > > > > > > > at org.junit.internal.runners.TestClassRunner.run ( > > > > > TestClassRunner.java:52) > > > > > > > > > > at junit.framework.JUnit4TestAdapter.run( > > > JUnit4TestAdapter.java > > > > > :32) > > > > > > > > > > > > > > > > > > > > AT 3,443,466 byes (or more) : > > > > > > > > > > > > > > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap > > > space > > > > > > > > > > at java.lang.AbstractStringBuilder.expandCapacity( > > > > > AbstractStringBuilder.java :99) > > > > > > > > > > at java.lang.AbstractStringBuilder.append ( > > > > > AbstractStringBuilder.java > > > > > :393) > > > > > > > > > > at java.lang.StringBuilder.append(StringBuilder.java :120) > > > > > > > > > > at org.junit.Assert.format(Assert.java:321) > > > > > > > > > > at org.junit.ComparisonFailure$ComparisonCompactor.compact( > > > > > ComparisonFailure.java:80) > > > > > > > > > > at org.junit.ComparisonFailure.getMessage( > > > ComparisonFailure.java > > > > > :37) > > > > > > > > > > at java.lang.Throwable.getLocalizedMessage(Throwable.java > > > :267) > > > > > > > > > > at java.lang.Throwable.toString (Throwable.java:344) > > > > > > > > > > at java.lang.String.valueOf(String.java:2615) > > > > > > > > > > at java.io.PrintWriter.print(PrintWriter.java:546) > > > > > > > > > > at java.io.PrintWriter.println(PrintWriter.java:683) > > > > > > > > > > at java.lang.Throwable.printStackTrace(Throwable.java:510) > > > > > > > > > > at org.apache.tools.ant.util.StringUtils.getStackTrace ( > > > > > StringUtils.java:96) > > > > > > > > > > at > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.getFilteredTrace > > > > > (JUnitTestRunner.java:856) > > > > > > > > > > at > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.formatError > > > > > (XMLJUnitResultFormatter.java:280) > > > > > > > > > > at > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.addError > > > > > (XMLJUnitResultFormatter.java:255) > > > > > > > > > > at > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner$4.addError( > > > > > JUnitTestRunner.java:988) > > > > > > > > > > at junit.framework.TestResult.addError(TestResult.java :38) > > > > > > > > > > at junit.framework.JUnit4TestAdapterCache$1.testFailure ( > > > > > JUnit4TestAdapterCache.java:51) > > > > > > > > > > at > org.junit.runner.notification.RunNotifier$4.notifyListener > > > ( > > > > > RunNotifier.java:96) > > > > > > > > > > at > org.junit.runner.notification.RunNotifier$SafeNotifier.run( > > > > > RunNotifier.java:37) > > > > > > > > > > at > org.junit.runner.notification.RunNotifier.fireTestFailure( > > > > > RunNotifier.java:93) > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.addFailure ( > > > > > TestMethodRunner.java:104) > > > > > > > > > > at > org.junit.internal.runners.TestMethodRunner.runUnprotected > > > ( > > > > > TestMethodRunner.java:87) > > > > > > > > > > at > > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected ( > > > > > BeforeAndAfterRunner.java:34) > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.runMethod( > > > > > TestMethodRunner.java:75) > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.run ( > > > > > TestMethodRunner.java :45) > > > > > > > > > > at > > > > > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod > ( > > > > > TestClassMethodsRunner.java:71) > > > > > > > > > > at org.junit.internal.runners.TestClassMethodsRunner.run( > > > > > TestClassMethodsRunner.java :35) > > > > > > > > > > at > > > org.junit.internal.runners.TestClassRunner$1.runUnprotected( > > > > > TestClassRunner.java :42) > > > > > > > > > > at > > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > > > > > BeforeAndAfterRunner.java:34) > > > > > > > > > > at org.junit.internal.runners.TestClassRunner.run ( > > > > > TestClassRunner.java:52) > > > > > > > > > > > > > > > I am writing a filesystem crawler so I need to be able to crawl > and > > > > > index > > > > > any size file (within reason). A 3-4MB file is certainly within > > > reason. > > > > > I > > > > > rewrote my code to store the file contents in a file and > read/write > > > in > > > > > one > > > > > line at a time. However, when I post the XML file to Solr using > > > > > SimplePostTool, I get another OutOfMemoryError about the java heap > > > space > > > > > > > > > > (thrown from org.xmlpull... again). In any case, does anyone have > > > any > > > > > ideas > > > > > about this? Has anyone posted documents with contents larger than > > > 3.5MBto > > > > > Solr successfully? If so, how was it done? I'm using Solr v1.2. > > > > > > > > > > > > > > > Best, > > > > > > > > > > Dave > > > > > > > > > > > > > > > > > > > > > > >
