Tika 1.6 has PDFBox 1.8.4, which has memory issues, eating excessive RAM! Either upgrade to Tika 1.7 (out now) or manually use the PDFBox 1.8.8 dependency.
M. On Friday 16 January 2015 15:21:55 Charlie Hull wrote: > On 16/01/2015 04:02, Dan Davis wrote: > > Why re-write all the document conversion in Java ;) Tika is very slow. > > 5 > > GB PDF is very big. > > Or you can run Tika in a separate process, or even on a separate > machine, wrapped with something to cope if it dies due to some horrible > input...we generally avoid document format translation within Solr and > do it externally before feeding documents to Solr. > > Charlie > > > If you have a lot of PDF like that try pdftotext in HTML and UTF-8 output > > mode. The HTML mode captures some meta-data that would otherwise be > > lost. > > > > > > If you need to go faster still, you can also write some stuff linked > > directly against poppler library. > > > > Before you jump down by through about Tika being slow - I wrote a PDF > > indexer that ran at 36 MB/s per core. Different indexer, all C, lots of > > getjmp/longjmp. But fast... > > > > On Thu, Jan 15, 2015 at 1:54 PM, <ganesh.ya...@sungard.com> wrote: > >> Siegfried and Michael Thank you for your replies and help. > >> > >> -----Original Message----- > >> From: Siegfried Goeschl [mailto:sgoes...@gmx.at] > >> Sent: Thursday, January 15, 2015 3:45 AM > >> To: solr-user@lucene.apache.org > >> Subject: Re: OutOfMemoryError for PDF document upload into Solr > >> > >> Hi Ganesh, > >> > >> you can increase the heap size but parsing a 4 GB PDF document will very > >> likely consume A LOT OF memory - I think you need to check if that large > >> PDF can be parsed at all :-) > >> > >> Cheers, > >> > >> Siegfried Goeschl > >> > >> On 14.01.15 18:04, Michael Della Bitta wrote: > >>> Yep, you'll have to increase the heap size for your Tomcat container. > >>> > >>> http://stackoverflow.com/questions/6897476/tomcat-7-how-to-set-initial > >>> -heap-size-correctly > >>> > >>> Michael Della Bitta > >>> > >>> Senior Software Engineer > >>> > >>> o: +1 646 532 3062 > >>> > >>> appinions inc. > >>> > >>> “The Science of Influence Marketing” > >>> > >>> 18 East 41st Street > >>> > >>> New York, NY 10017 > >>> > >>> t: @appinions <https://twitter.com/Appinions> | g+: > >>> plus.google.com/appinions > >>> <https://plus.google.com/u/0/b/112002776285509593336/11200277628550959 > >>> 3336/posts> > >>> w: appinions.com <http://www.appinions.com/> > >>> > >>> On Wed, Jan 14, 2015 at 12:00 PM, <ganesh.ya...@sungard.com> wrote: > >>>> Hello, > >>>> > >>>> Can someone pass on the hints to get around following error? Is there > >>>> any Heap Size parameter I can set in Tomcat or in Solr webApp that > >>>> gets deployed in Solr? > >>>> > >>>> I am running Solr webapp inside Tomcat on my local machine which has > >>>> RAM of 12 GB. I have PDF document which is 4 GB max in size that > >>>> needs to be loaded into Solr > >>>> > >>>> > >>>> > >>>> > >>>> Exception in thread "http-apr-8983-exec-6" java.lang. : Java heap > >> > >> space > >> > >>>> at java.util.AbstractCollection.toArray(Unknown Source) > >>>> at java.util.ArrayList.<init>(Unknown Source) > >>>> at > >>>> > >>>> org.apache.pdfbox.cos.COSDocument.getObjects(COSDocument.java:518) > >>>> > >>>> at > >> > >> org.apache.pdfbox.cos.COSDocument.close(COSDocument.java:575) > >> > >>>> at > >> > >> org.apache.pdfbox.pdfparser.PDFParser.parse(PDFParser.java:254) > >> > >>>> at > >> > >> org.apache.pdfbox.pdmodel.PDDocument.load(PDDocument.java:1238) > >> > >>>> at > >> > >> org.apache.pdfbox.pdmodel.PDDocument.load(PDDocument.java:1203) > >> > >>>> at > >> > >> org.apache.tika.parser.pdf.PDFParser.parse(PDFParser.java:111) > >> > >>>> at > >>>> > >>>> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242) > >>>> > >>>> at > >>>> > >>>> org.apache.tika.parser.CompositeParser.parse(CompositeParser.java:242) > >>>> > >>>> at > >>>> > >>>> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:120 > >>>> ) > >>>> > >>>> at > >> > >> org.apache.solr.handler.extraction.ExtractingDocumentLoader.load(Extracti > >> ngDocumentLoader.java:219)>> > >>>> at > >> > >> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(Conten > >> tStreamHandlerBase.java:74)>> > >>>> at > >> > >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBa > >> se.java:135)>> > >>>> at > >> > >> org.apache.solr.core.RequestHandlers$LazyRequestHandlerWrapper.handleRequ > >> est(RequestHandlers.java:246)>> > >>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967) > >>>> at > >> > >> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.jav > >> a:777)>> > >>>> at > >> > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.ja > >> va:418)>> > >>>> at > >> > >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.ja > >> va:207)>> > >>>> at > >> > >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicat > >> ionFilterChain.java:241)>> > >>>> at > >> > >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilte > >> rChain.java:208)>> > >>>> at > >> > >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve > >> .java:220)>> > >>>> at > >> > >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve > >> .java:122)>> > >>>> at > >> > >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java: > >> 170)>> > >>>> at > >> > >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java: > >> 103)>> > >>>> at > >> > >> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) > >> > >>>> at > >> > >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.j > >> ava:116)>> > >>>> at > >> > >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:42 > >> 1) > >> > >>>> at > >> > >> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Pr > >> ocessor.java:1070)>> > >>>> at > >> > >> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abst > >> ractProtocol.java:611)>> > >>>> at > >> > >> org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint. > >> java:2462)>> > >>>> at > >>>> > >>>> org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoin > >>>> t.java:2451) > >>>> > >>>> > >>>> Thanks > >>>> Ganesh
