I don't think this is a StringBuilder limitation, but rather your Java JVM doesn't start with enough memory. i.e. -Xmx.
In raw Lucene, I've indexed 240M files........ Best Erick On Jan 16, 2008 10:12 AM, David Thibault <[EMAIL PROTECTED]> wrote: > All, > I just found a thread about this on the mailing list archives because I'm > troubleshooting the same problem. The kicker is that it doesn't take such > large files to kill the StringBuilder. I have discovered the following: > > By using a text file made up of 3,443,464 bytes or less, I get no error. > > AT 3,443,465 bytes: > > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > > at java.lang.String.<init>(String.java:208) > > at java.lang.StringBuilder.toString(StringBuilder.java:431) > > at org.junit.Assert.format(Assert.java:321) > > at org.junit.ComparisonFailure$ComparisonCompactor.compact( > ComparisonFailure.java:80) > > at org.junit.ComparisonFailure.getMessage(ComparisonFailure.java > :37) > > at java.lang.Throwable.getLocalizedMessage(Throwable.java:267) > > at java.lang.Throwable.toString(Throwable.java:344) > > at java.lang.String.valueOf(String.java:2615) > > at java.io.PrintWriter.print(PrintWriter.java:546) > > at java.io.PrintWriter.println(PrintWriter.java:683) > > at java.lang.Throwable.printStackTrace(Throwable.java:510) > > at org.apache.tools.ant.util.StringUtils.getStackTrace( > StringUtils.java:96) > > at > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.getFilteredTrace > (JUnitTestRunner.java:856) > > at > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.formatError > (XMLJUnitResultFormatter.java:280) > > at > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.addError > (XMLJUnitResultFormatter.java:255) > > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner$4.addError( > JUnitTestRunner.java:988) > > at junit.framework.TestResult.addError(TestResult.java:38) > > at junit.framework.JUnit4TestAdapterCache$1.testFailure( > JUnit4TestAdapterCache.java:51) > > at org.junit.runner.notification.RunNotifier$4.notifyListener( > RunNotifier.java:96) > > at org.junit.runner.notification.RunNotifier$SafeNotifier.run( > RunNotifier.java:37) > > at org.junit.runner.notification.RunNotifier.fireTestFailure( > RunNotifier.java:93) > > at org.junit.internal.runners.TestMethodRunner.addFailure( > TestMethodRunner.java:104) > > at org.junit.internal.runners.TestMethodRunner.runUnprotected( > TestMethodRunner.java:87) > > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > BeforeAndAfterRunner.java:34) > > at org.junit.internal.runners.TestMethodRunner.runMethod( > TestMethodRunner.java:75) > > at org.junit.internal.runners.TestMethodRunner.run( > TestMethodRunner.java:45) > > at > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod( > TestClassMethodsRunner.java:71) > > at org.junit.internal.runners.TestClassMethodsRunner.run( > TestClassMethodsRunner.java:35) > > at org.junit.internal.runners.TestClassRunner$1.runUnprotected( > TestClassRunner.java:42) > > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > BeforeAndAfterRunner.java:34) > > at org.junit.internal.runners.TestClassRunner.run( > TestClassRunner.java:52) > > at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:32) > > > > AT 3,443,466 byes (or more) : > > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > > at java.lang.AbstractStringBuilder.expandCapacity( > AbstractStringBuilder.java:99) > > at java.lang.AbstractStringBuilder.append( > AbstractStringBuilder.java > :393) > > at java.lang.StringBuilder.append(StringBuilder.java:120) > > at org.junit.Assert.format(Assert.java:321) > > at org.junit.ComparisonFailure$ComparisonCompactor.compact( > ComparisonFailure.java:80) > > at org.junit.ComparisonFailure.getMessage(ComparisonFailure.java > :37) > > at java.lang.Throwable.getLocalizedMessage(Throwable.java:267) > > at java.lang.Throwable.toString(Throwable.java:344) > > at java.lang.String.valueOf(String.java:2615) > > at java.io.PrintWriter.print(PrintWriter.java:546) > > at java.io.PrintWriter.println(PrintWriter.java:683) > > at java.lang.Throwable.printStackTrace(Throwable.java:510) > > at org.apache.tools.ant.util.StringUtils.getStackTrace( > StringUtils.java:96) > > at > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.getFilteredTrace > (JUnitTestRunner.java:856) > > at > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.formatError > (XMLJUnitResultFormatter.java:280) > > at > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.addError > (XMLJUnitResultFormatter.java:255) > > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner$4.addError( > JUnitTestRunner.java:988) > > at junit.framework.TestResult.addError(TestResult.java:38) > > at junit.framework.JUnit4TestAdapterCache$1.testFailure( > JUnit4TestAdapterCache.java:51) > > at org.junit.runner.notification.RunNotifier$4.notifyListener( > RunNotifier.java:96) > > at org.junit.runner.notification.RunNotifier$SafeNotifier.run( > RunNotifier.java:37) > > at org.junit.runner.notification.RunNotifier.fireTestFailure( > RunNotifier.java:93) > > at org.junit.internal.runners.TestMethodRunner.addFailure( > TestMethodRunner.java:104) > > at org.junit.internal.runners.TestMethodRunner.runUnprotected( > TestMethodRunner.java:87) > > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > BeforeAndAfterRunner.java:34) > > at org.junit.internal.runners.TestMethodRunner.runMethod( > TestMethodRunner.java:75) > > at org.junit.internal.runners.TestMethodRunner.run( > TestMethodRunner.java:45) > > at > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod( > TestClassMethodsRunner.java:71) > > at org.junit.internal.runners.TestClassMethodsRunner.run( > TestClassMethodsRunner.java:35) > > at org.junit.internal.runners.TestClassRunner$1.runUnprotected( > TestClassRunner.java:42) > > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > BeforeAndAfterRunner.java:34) > > at org.junit.internal.runners.TestClassRunner.run( > TestClassRunner.java:52) > > > I am writing a filesystem crawler so I need to be able to crawl and index > any size file (within reason). A 3-4MB file is certainly within reason. > I > rewrote my code to store the file contents in a file and read/write in one > line at a time. However, when I post the XML file to Solr using > SimplePostTool, I get another OutOfMemoryError about the java heap space > (thrown from org.xmlpull... again). In any case, does anyone have any > ideas > about this? Has anyone posted documents with contents larger than 3.5MBto > Solr successfully? If so, how was it done? I'm using Solr v1.2. > > > Best, > > Dave >
