I think your biggest problem is requesting 70,000 records from Solr. That is not going to be fast.
Two suggestions: 1. Use paging. Get the results in chunks, 10, 25, 100, whatever. 2. Since you are running out of memory parsing XML, I'm guessing that you're using a DOM-style parser. Don't do that. You do not need to create elaborate structures, strip mine the data, then throw those structures away. Instead, us a streaming parser, like Stax. The sounds like an XY problem. What are you trying to achieve by fetching 10,000 records? There is probably a better way to do it. wunder On 12/12/07 11:58 AM, "Owens, Martin" <[EMAIL PROTECTED]> wrote: > Hello everyone, > > I'm looking for a better solution that the current xml output we're currently > getting; if you return more than 70k records the webserver can no longer cope > with parsing the xml and the machine falls over out of memory. > > Ideally what we'd like is for the search results to go directly into a > temporary mysql table so we can link against it in a further request from the > web server. Does anyone know any plugs or people who have done anything along > these lines? > > We might be able to settle for receiving the single field column as a csv type > file, that would at least let us cut down on the processing and parsing. I see > there is a csv indexer but do we have a csv output plugin? > > Once again thank you all for your help. > > Best Regards, Martin Ownes
