I guess it would be fater if you run this program from a diffferent box,.that way the disk IO will not be a bottleneck
2009/7/10 Noble Paul നോബിള് नोब्ळ् <noble.p...@corp.aol.com>: > are these xml files in the solr add xml format? > > When you post using curl, I guess it opens as many http connections as > there are files. if you can write a small program to post all these > files in one request, you should be able to get better perf. > > the following can be the pseudo-code > > open connection > write "<root>" > for each file > write filecontent > write "</root>" > close connection > > > > > On Fri, Jul 10, 2009 at 10:23 AM, Francis Yakin<fya...@liquid.com> wrote: >> >> I have about 1000 folders, each folder consist 2581 xml files. Total of xml >> files is ~ 2.6 millions >> >> I developed perl script, inside my script it's executed this cmd: >> >> curl http://localhost:7001/solr/update --data-binary "@0039000.xml" -H >> 'Content-type:text/plain; charset=utf-8' >> >> It tooks me about 4 1/2 hrs to load and commit. >> >> I would like to know the advantages using curl to posting/add/update the xml >> files to solr comparing with using WebService::Solr module? >> >> Is using WebService::Solr faster? >> >> The XML files are local on the Solr Master box, so I posting it locally( not >> using wan or lan). >> >> Any input will be much appreciated. >> >> Thanks >> >> Francis >> >> >> > > > > -- > ----------------------------------------------------- > Noble Paul | Principal Engineer| AOL | http://aol.com > -- ----------------------------------------------------- Noble Paul | Principal Engineer| AOL | http://aol.com
