I think your PS might do the trick. My JVM doesn't seem to be the issue, because I've set it to -Xmx512m -Xms256m. I will track down the solr config parameter you mentioned and try that. Thanks for the quick response! Dave
On 1/16/08, Erick Erickson <[EMAIL PROTECTED]> wrote: > > P.S. Lucene by default limits the maximum field length > to 10K tokens, so you have to bump that for large files. > > Erick > > On Jan 16, 2008 11:04 AM, Erick Erickson <[EMAIL PROTECTED]> wrote: > > > I don't think this is a StringBuilder limitation, but rather your Java > > JVM doesn't start with enough memory. i.e. -Xmx. > > > > In raw Lucene, I've indexed 240M files........ > > > > Best > > Erick > > > > > > On Jan 16, 2008 10:12 AM, David Thibault <[EMAIL PROTECTED]> > > wrote: > > > > > All, > > > I just found a thread about this on the mailing list archives because > > > I'm > > > troubleshooting the same problem. The kicker is that it doesn't take > > > such > > > large files to kill the StringBuilder. I have discovered the > following: > > > > > > > > > By using a text file made up of 3,443,464 bytes or less, I get no > > > error. > > > > > > AT 3,443,465 bytes: > > > > > > > > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > > > > > > at java.lang.String .<init>(String.java:208) > > > > > > at java.lang.StringBuilder.toString(StringBuilder.java:431) > > > > > > at org.junit.Assert.format(Assert.java:321) > > > > > > at org.junit.ComparisonFailure$ComparisonCompactor.compact ( > > > ComparisonFailure.java:80) > > > > > > at org.junit.ComparisonFailure.getMessage( > ComparisonFailure.java > > > :37) > > > > > > at java.lang.Throwable.getLocalizedMessage(Throwable.java:267) > > > > > > at java.lang.Throwable.toString (Throwable.java:344) > > > > > > at java.lang.String.valueOf(String.java:2615) > > > > > > at java.io.PrintWriter.print(PrintWriter.java:546) > > > > > > at java.io.PrintWriter.println(PrintWriter.java:683) > > > > > > at java.lang.Throwable.printStackTrace(Throwable.java:510) > > > > > > at org.apache.tools.ant.util.StringUtils.getStackTrace( > > > StringUtils.java:96) > > > > > > at > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.getFilteredTrace > > > (JUnitTestRunner.java:856) > > > > > > at > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.formatError > > > (XMLJUnitResultFormatter.java:280) > > > > > > at > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.addError > > > (XMLJUnitResultFormatter.java:255) > > > > > > at > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner$4.addError( > > > JUnitTestRunner.java:988) > > > > > > at junit.framework.TestResult.addError(TestResult.java :38) > > > > > > at junit.framework.JUnit4TestAdapterCache$1.testFailure( > > > JUnit4TestAdapterCache.java:51) > > > > > > at org.junit.runner.notification.RunNotifier$4.notifyListener( > > > RunNotifier.java:96) > > > > > > at org.junit.runner.notification.RunNotifier$SafeNotifier.run( > > > RunNotifier.java:37) > > > > > > at org.junit.runner.notification.RunNotifier.fireTestFailure( > > > RunNotifier.java:93) > > > > > > at org.junit.internal.runners.TestMethodRunner.addFailure ( > > > TestMethodRunner.java:104) > > > > > > at org.junit.internal.runners.TestMethodRunner.runUnprotected( > > > TestMethodRunner.java:87) > > > > > > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected > ( > > > BeforeAndAfterRunner.java:34) > > > > > > at org.junit.internal.runners.TestMethodRunner.runMethod( > > > TestMethodRunner.java:75) > > > > > > at org.junit.internal.runners.TestMethodRunner.run( > > > TestMethodRunner.java :45) > > > > > > at > > > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod( > > > TestClassMethodsRunner.java:71) > > > > > > at org.junit.internal.runners.TestClassMethodsRunner.run( > > > TestClassMethodsRunner.java :35) > > > > > > at org.junit.internal.runners.TestClassRunner$1.runUnprotected( > > > TestClassRunner.java:42) > > > > > > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected > ( > > > BeforeAndAfterRunner.java:34) > > > > > > at org.junit.internal.runners.TestClassRunner.run( > > > TestClassRunner.java:52) > > > > > > at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java > > > :32) > > > > > > > > > > > > AT 3,443,466 byes (or more) : > > > > > > > > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap space > > > > > > at java.lang.AbstractStringBuilder.expandCapacity( > > > AbstractStringBuilder.java:99) > > > > > > at java.lang.AbstractStringBuilder.append ( > > > AbstractStringBuilder.java > > > :393) > > > > > > at java.lang.StringBuilder.append(StringBuilder.java:120) > > > > > > at org.junit.Assert.format(Assert.java:321) > > > > > > at org.junit.ComparisonFailure$ComparisonCompactor.compact ( > > > ComparisonFailure.java:80) > > > > > > at org.junit.ComparisonFailure.getMessage( > ComparisonFailure.java > > > :37) > > > > > > at java.lang.Throwable.getLocalizedMessage(Throwable.java:267) > > > > > > at java.lang.Throwable.toString (Throwable.java:344) > > > > > > at java.lang.String.valueOf(String.java:2615) > > > > > > at java.io.PrintWriter.print(PrintWriter.java:546) > > > > > > at java.io.PrintWriter.println(PrintWriter.java:683) > > > > > > at java.lang.Throwable.printStackTrace(Throwable.java:510) > > > > > > at org.apache.tools.ant.util.StringUtils.getStackTrace( > > > StringUtils.java:96) > > > > > > at > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.getFilteredTrace > > > (JUnitTestRunner.java:856) > > > > > > at > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.formatError > > > (XMLJUnitResultFormatter.java:280) > > > > > > at > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.addError > > > (XMLJUnitResultFormatter.java:255) > > > > > > at > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner$4.addError( > > > JUnitTestRunner.java:988) > > > > > > at junit.framework.TestResult.addError(TestResult.java :38) > > > > > > at junit.framework.JUnit4TestAdapterCache$1.testFailure( > > > JUnit4TestAdapterCache.java:51) > > > > > > at org.junit.runner.notification.RunNotifier$4.notifyListener( > > > RunNotifier.java:96) > > > > > > at org.junit.runner.notification.RunNotifier$SafeNotifier.run( > > > RunNotifier.java:37) > > > > > > at org.junit.runner.notification.RunNotifier.fireTestFailure( > > > RunNotifier.java:93) > > > > > > at org.junit.internal.runners.TestMethodRunner.addFailure ( > > > TestMethodRunner.java:104) > > > > > > at org.junit.internal.runners.TestMethodRunner.runUnprotected( > > > TestMethodRunner.java:87) > > > > > > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected > ( > > > BeforeAndAfterRunner.java:34) > > > > > > at org.junit.internal.runners.TestMethodRunner.runMethod( > > > TestMethodRunner.java:75) > > > > > > at org.junit.internal.runners.TestMethodRunner.run( > > > TestMethodRunner.java :45) > > > > > > at > > > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod( > > > TestClassMethodsRunner.java:71) > > > > > > at org.junit.internal.runners.TestClassMethodsRunner.run( > > > TestClassMethodsRunner.java :35) > > > > > > at org.junit.internal.runners.TestClassRunner$1.runUnprotected( > > > TestClassRunner.java:42) > > > > > > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected > ( > > > BeforeAndAfterRunner.java:34) > > > > > > at org.junit.internal.runners.TestClassRunner.run( > > > TestClassRunner.java:52) > > > > > > > > > I am writing a filesystem crawler so I need to be able to crawl and > > > index > > > any size file (within reason). A 3-4MB file is certainly within > reason. > > > I > > > rewrote my code to store the file contents in a file and read/write in > > > one > > > line at a time. However, when I post the XML file to Solr using > > > SimplePostTool, I get another OutOfMemoryError about the java heap > space > > > > > > (thrown from org.xmlpull... again). In any case, does anyone have any > > > ideas > > > about this? Has anyone posted documents with contents larger than > 3.5MBto > > > Solr successfully? If so, how was it done? I'm using Solr v1.2. > > > > > > > > > Best, > > > > > > Dave > > > > > > > >
