Nice signature...=)
On 1/16/08, Erick Erickson <[EMAIL PROTECTED]> wrote: > > The PS really wasn't related to your OOM, and raising that shouldn't > have changed the behavior. All that happens if you go beyond 10,000 > tokens is that the rest gets thrown away. > > But we're beyond my real knowledge level about SOLR, so I'll defer > to others. A very quick-n-dirty test as to whether you're actually > allocation more memory to the process you *think* you are would be > to bump it ridiculously higher. I'm completely unclear about what > process gets the increased memory relative to the server. > > [EMAIL PROTECTED] > > > On Jan 16, 2008 11:33 AM, David Thibault <[EMAIL PROTECTED]> > wrote: > > > I tried raising the <maxFieldLength>100000000</maxFieldLength> under > > <mainIndex> as well as <indexDefaults> and still no luck. I'm trying to > > upload a text file that is about 8 MB in size. I think the following > > stack > > trace still points to some sort of overflowed String issue. Thoughts? > > Solr returned an error: Java heap space java.lang.OutOfMemoryError: > Java > > heap space > > at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232) > > at java.lang.StringCoding.encode(StringCoding.java:272) > > at java.lang.String.getBytes(String.java:947) > > at org.apache.lucene.index.FieldsWriter.addDocument(FieldsWriter.java > :98) > > at org.apache.lucene.index.DocumentWriter.addDocument( > DocumentWriter.java > > :107) > > > > at org.apache.lucene.index.IndexWriter.buildSingleDocSegment( > > IndexWriter.java:977) > > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:965) > > at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:947) > > at org.apache.solr.update.DirectUpdateHandler2.addDoc( > > DirectUpdateHandler2.java:270) > > at org.apache.solr.handler.XmlUpdateRequestHandler.update( > > XmlUpdateRequestHandler.java:166) > > at org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody( > > XmlUpdateRequestHandler.java:84) > > at org.apache.solr.handler.RequestHandlerBase.handleRequest( > > RequestHandlerBase.java:77) > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:658) at > > org.apache.solr.servlet.SolrDispatchFilter.execute( > SolrDispatchFilter.java > > :191) > > > > at org.apache.solr.servlet.SolrDispatchFilter.doFilter( > > SolrDispatchFilter.java:159) > > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter( > > ApplicationFilterChain.java:215) > > at org.apache.catalina.core.ApplicationFilterChain.doFilter( > > ApplicationFilterChain.java:188) > > at org.apache.catalina.core.StandardWrapperValve.invoke( > > StandardWrapperValve.java:213) > > at org.apache.catalina.core.StandardContextValve.invoke( > > StandardContextValve.java:174) > > at org.apache.catalina.core.StandardHostValve.invoke( > > StandardHostValve.java > > :127) > > at org.apache.catalina.valves.ErrorReportValve.invoke( > > ErrorReportValve.java > > :117) > > at org.apache.catalina.core.StandardEngineValve.invoke( > > StandardEngineValve.java:108) > > at org.apache.catalina.connector.CoyoteAdapter.service( > CoyoteAdapter.java > > :151) > > > > at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java > > :874) > > > > at > > > > > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection > > (Http11BaseProtocol.java:665) > > at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket( > > PoolTcpEndpoint.java:528) > > at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt( > > LeaderFollowerWorkerThread.java:81) > > at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run( > > ThreadPool.java:689) > > at java.lang.Thread.run(Thread.java:619) > > > > java.io.IOException: Server returned HTTP response code: 500 for URL: > > http://solr:8080/solr/update > > at sun.net.www.protocol.http.HttpURLConnection.getInputStream( > > HttpURLConnection.java:1170) > > at > com.itstrategypartners.sents.solrUpload.SimplePostTool.postData( > > SimplePostTool.java:134) > > at > com.itstrategypartners.sents.solrUpload.SimplePostTool.postFile( > > SimplePostTool.java:87) > > at com.itstrategypartners.sents.solrUpload.Uploader.uploadFile( > > Uploader.java:97) > > at > com.itstrategypartners.sents.solrUpload.UploaderTest.uploadFile( > > UploaderTest.java:95) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at sun.reflect.NativeMethodAccessorImpl.invoke( > > NativeMethodAccessorImpl.java:39) > > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > > DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:585) > > at org.junit.internal.runners.TestMethodRunner.executeMethodBody( > > TestMethodRunner.java:99) > > at org.junit.internal.runners.TestMethodRunner.runUnprotected( > > TestMethodRunner.java:81) > > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > > BeforeAndAfterRunner.java:34) > > at org.junit.internal.runners.TestMethodRunner.runMethod( > > TestMethodRunner.java:75) > > at org.junit.internal.runners.TestMethodRunner.run( > > TestMethodRunner.java:45) > > at > > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod( > > TestClassMethodsRunner.java:71) > > at org.junit.internal.runners.TestClassMethodsRunner.run( > > TestClassMethodsRunner.java:35) > > at org.junit.internal.runners.TestClassRunner$1.runUnprotected( > > TestClassRunner.java:42) > > at org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > > BeforeAndAfterRunner.java:34) > > at org.junit.internal.runners.TestClassRunner.run( > > TestClassRunner.java:52) > > at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java > :32) > > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run > > ( > > JUnitTestRunner.java:421) > > at > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch( > > JUnitTestRunner.java:912) > > at > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main > > (JUnitTestRunner.java:766) > > > > On 1/16/08, David Thibault <[EMAIL PROTECTED]> wrote: > > > > > > I think your PS might do the trick. My JVM doesn't seem to be the > > issue, > > > because I've set it to -Xmx512m -Xms256m. I will track down the solr > > config > > > parameter you mentioned and try that. Thanks for the quick response! > > > > > > Dave > > > > > > On 1/16/08, Erick Erickson <[EMAIL PROTECTED] > wrote: > > > > > > > > P.S. Lucene by default limits the maximum field length > > > > to 10K tokens, so you have to bump that for large files. > > > > > > > > Erick > > > > > > > > On Jan 16, 2008 11:04 AM, Erick Erickson <[EMAIL PROTECTED]> > > > > wrote: > > > > > > > > > I don't think this is a StringBuilder limitation, but rather your > > Java > > > > > > > > > JVM doesn't start with enough memory. i.e. -Xmx. > > > > > > > > > > In raw Lucene, I've indexed 240M files........ > > > > > > > > > > Best > > > > > Erick > > > > > > > > > > > > > > > On Jan 16, 2008 10:12 AM, David Thibault < > > [EMAIL PROTECTED] > > > > > > > > > > wrote: > > > > > > > > > > > All, > > > > > > I just found a thread about this on the mailing list archives > > > > because > > > > > > I'm > > > > > > troubleshooting the same problem. The kicker is that it doesn't > > > > take > > > > > > such > > > > > > large files to kill the StringBuilder. I have discovered the > > > > following: > > > > > > > > > > > > > > > > > > By using a text file made up of 3,443,464 bytes or less, I get > no > > > > > > error. > > > > > > > > > > > > AT 3,443,465 bytes: > > > > > > > > > > > > > > > > > > Exception in thread "main" java.lang.OutOfMemoryError : Java > heap > > > > space > > > > > > > > > > > > at java.lang.String .<init>(String.java:208) > > > > > > > > > > > > at java.lang.StringBuilder.toString(StringBuilder.java > :431) > > > > > > > > > > > > at org.junit.Assert.format(Assert.java:321) > > > > > > > > > > > > at > org.junit.ComparisonFailure$ComparisonCompactor.compact( > > > > > > ComparisonFailure.java:80) > > > > > > > > > > > > at org.junit.ComparisonFailure.getMessage ( > > > > ComparisonFailure.java > > > > > > :37) > > > > > > > > > > > > at java.lang.Throwable.getLocalizedMessage(Throwable.java > > > > :267) > > > > > > > > > > > > at java.lang.Throwable.toString (Throwable.java:344) > > > > > > > > > > > > at java.lang.String.valueOf(String.java:2615) > > > > > > > > > > > > at java.io.PrintWriter.print(PrintWriter.java:546) > > > > > > > > > > > > at java.io.PrintWriter.println (PrintWriter.java:683) > > > > > > > > > > > > at java.lang.Throwable.printStackTrace(Throwable.java > :510) > > > > > > > > > > > > at org.apache.tools.ant.util.StringUtils.getStackTrace( > > > > > > StringUtils.java :96) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.getFilteredTrace > > > > > > (JUnitTestRunner.java:856) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.formatError > > > > > > (XMLJUnitResultFormatter.java:280) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.addError > > > > > > (XMLJUnitResultFormatter.java:255) > > > > > > > > > > > > at > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner$4.addError( > > > > > > JUnitTestRunner.java:988) > > > > > > > > > > > > at junit.framework.TestResult.addError(TestResult.java:38) > > > > > > > > > > > > at junit.framework.JUnit4TestAdapterCache$1.testFailure( > > > > > > JUnit4TestAdapterCache.java:51) > > > > > > > > > > > > at > > org.junit.runner.notification.RunNotifier$4.notifyListener > > > > ( > > > > > > RunNotifier.java:96) > > > > > > > > > > > > at > > org.junit.runner.notification.RunNotifier$SafeNotifier.run > > > > ( > > > > > > RunNotifier.java:37) > > > > > > > > > > > > at > > org.junit.runner.notification.RunNotifier.fireTestFailure( > > > > > > RunNotifier.java:93) > > > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.addFailure( > > > > > > TestMethodRunner.java:104) > > > > > > > > > > > > at > > org.junit.internal.runners.TestMethodRunner.runUnprotected > > > > ( > > > > > > TestMethodRunner.java:87) > > > > > > > > > > > > at > > > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected ( > > > > > > BeforeAndAfterRunner.java:34) > > > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.runMethod( > > > > > > TestMethodRunner.java:75) > > > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.run ( > > > > > > TestMethodRunner.java :45) > > > > > > > > > > > > at > > > > > > > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod > > ( > > > > > > TestClassMethodsRunner.java:71) > > > > > > > > > > > > at org.junit.internal.runners.TestClassMethodsRunner.run( > > > > > > TestClassMethodsRunner.java :35) > > > > > > > > > > > > at > > > > org.junit.internal.runners.TestClassRunner$1.runUnprotected( > > > > > > TestClassRunner.java :42) > > > > > > > > > > > > at > > > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > > > > > > BeforeAndAfterRunner.java:34) > > > > > > > > > > > > at org.junit.internal.runners.TestClassRunner.run ( > > > > > > TestClassRunner.java:52) > > > > > > > > > > > > at junit.framework.JUnit4TestAdapter.run( > > > > JUnit4TestAdapter.java > > > > > > :32) > > > > > > > > > > > > > > > > > > > > > > > > AT 3,443,466 byes (or more) : > > > > > > > > > > > > > > > > > > Exception in thread "main" java.lang.OutOfMemoryError: Java heap > > > > space > > > > > > > > > > > > at java.lang.AbstractStringBuilder.expandCapacity( > > > > > > AbstractStringBuilder.java :99) > > > > > > > > > > > > at java.lang.AbstractStringBuilder.append ( > > > > > > AbstractStringBuilder.java > > > > > > :393) > > > > > > > > > > > > at java.lang.StringBuilder.append(StringBuilder.java:120) > > > > > > > > > > > > at org.junit.Assert.format(Assert.java:321) > > > > > > > > > > > > at > org.junit.ComparisonFailure$ComparisonCompactor.compact( > > > > > > ComparisonFailure.java:80) > > > > > > > > > > > > at org.junit.ComparisonFailure.getMessage( > > > > ComparisonFailure.java > > > > > > :37) > > > > > > > > > > > > at java.lang.Throwable.getLocalizedMessage(Throwable.java > > > > :267) > > > > > > > > > > > > at java.lang.Throwable.toString (Throwable.java:344) > > > > > > > > > > > > at java.lang.String.valueOf(String.java:2615) > > > > > > > > > > > > at java.io.PrintWriter.print(PrintWriter.java:546) > > > > > > > > > > > > at java.io.PrintWriter.println(PrintWriter.java:683) > > > > > > > > > > > > at java.lang.Throwable.printStackTrace(Throwable.java > :510) > > > > > > > > > > > > at org.apache.tools.ant.util.StringUtils.getStackTrace ( > > > > > > StringUtils.java:96) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.getFilteredTrace > > > > > > (JUnitTestRunner.java:856) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.formatError > > > > > > (XMLJUnitResultFormatter.java:280) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.XMLJUnitResultFormatter.addError > > > > > > (XMLJUnitResultFormatter.java:255) > > > > > > > > > > > > at > > > > > > > > > > > > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner$4.addError( > > > > > > JUnitTestRunner.java:988) > > > > > > > > > > > > at junit.framework.TestResult.addError(TestResult.java:38) > > > > > > > > > > > > at junit.framework.JUnit4TestAdapterCache$1.testFailure ( > > > > > > JUnit4TestAdapterCache.java:51) > > > > > > > > > > > > at > > org.junit.runner.notification.RunNotifier$4.notifyListener > > > > ( > > > > > > RunNotifier.java:96) > > > > > > > > > > > > at > > org.junit.runner.notification.RunNotifier$SafeNotifier.run( > > > > > > RunNotifier.java:37) > > > > > > > > > > > > at > > org.junit.runner.notification.RunNotifier.fireTestFailure( > > > > > > RunNotifier.java:93) > > > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.addFailure( > > > > > > TestMethodRunner.java:104) > > > > > > > > > > > > at > > org.junit.internal.runners.TestMethodRunner.runUnprotected > > > > ( > > > > > > TestMethodRunner.java:87) > > > > > > > > > > > > at > > > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected ( > > > > > > BeforeAndAfterRunner.java:34) > > > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.runMethod( > > > > > > TestMethodRunner.java:75) > > > > > > > > > > > > at org.junit.internal.runners.TestMethodRunner.run ( > > > > > > TestMethodRunner.java :45) > > > > > > > > > > > > at > > > > > > > org.junit.internal.runners.TestClassMethodsRunner.invokeTestMethod > > ( > > > > > > TestClassMethodsRunner.java:71) > > > > > > > > > > > > at org.junit.internal.runners.TestClassMethodsRunner.run( > > > > > > TestClassMethodsRunner.java :35) > > > > > > > > > > > > at > > > > org.junit.internal.runners.TestClassRunner$1.runUnprotected( > > > > > > TestClassRunner.java :42) > > > > > > > > > > > > at > > > > org.junit.internal.runners.BeforeAndAfterRunner.runProtected( > > > > > > BeforeAndAfterRunner.java:34) > > > > > > > > > > > > at org.junit.internal.runners.TestClassRunner.run ( > > > > > > TestClassRunner.java:52) > > > > > > > > > > > > > > > > > > I am writing a filesystem crawler so I need to be able to crawl > > and > > > > > > index > > > > > > any size file (within reason). A 3-4MB file is certainly within > > > > reason. > > > > > > I > > > > > > rewrote my code to store the file contents in a file and > > read/write > > > > in > > > > > > one > > > > > > line at a time. However, when I post the XML file to Solr using > > > > > > SimplePostTool, I get another OutOfMemoryError about the java > heap > > > > space > > > > > > > > > > > > (thrown from org.xmlpull... again). In any case, does anyone > have > > > > any > > > > > > ideas > > > > > > about this? Has anyone posted documents with contents larger > than > > > > 3.5MBto > > > > > > Solr successfully? If so, how was it done? I'm using Solr v1.2 > . > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Dave > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
