Well, this usually means the maximum packet size has been exceeded, there are several possibilities here that I'm going to skip over because I have to ask the purpose of indexing a 5G file.
Indexing such a huge file has several problems from a user's perspective: 1> assuming the bulk of it is text, it'll be hit on many, many searches. 2> because it is so large, it'll probably rank quite a ways down the list, users may rarely see it. 3> even if it is found and a user clicks on the doc, what then? You can't reasonably fetch it from a server and display it. In short, before diving into the mechanics of why you get this error and correcting that, I'd be sure it made any sense to even try to index this doc. It may, don't get me wrong. Just askin'. Also, if this is some kind of binary file (say a movie or something), and what you're trying to index is actually the metadata, consider extracting that on the client side with SolrJ and/or Tika and just sending the data you expect to index to Solr. This scales much better than sending huge does to Solr and letting that poor little server extract _and_ index _and_ serve queries ;). Best, Erick On Mon, Dec 14, 2015 at 9:04 AM, Antelmo Aguilar <antelmo.aguilar...@nd.edu> wrote: > Hello, > > I am trying to index a very large file in Solr (around 5GB). However, I > get out of memory errors using Curl. I tried using the post script and I > had some success with it. After indexing several hundred thousand records > though, I got the following error message: > > *SimplePostTool: FATAL: IOException while posting data: > java.io.IOException: too many bytes written* > > Would it be possible to get some help on where I can start looking to solve > this issue? I tried finding some type of log that would give me more > information. I have not had any luck. The only logs I was able to find > related to this error were the logs from Solr, but I assume these are from > the "server" perspective and not "cient's" perspective of the error. I > would really appreciate the help. > > Thanks, > Antelmo
