Yes, the xml files are in complete add format.
This is my code:
#!/usr/bin/perl
if (($#ARGV + 1) <= 0 ) {
print "Usage: perl prod.pl <dir> \n\n";
exit(1);
}
###### -- CHANGE accordingly
$timeout = 300;
$topdir = "/opt/Test/xml-file/";
#$topdir = "/opt/Test/";
$dir = $topdir . $ARGV[0];
$commit_dir = "/opt/commit";
#####
$curl="/usr/bin/curl";
print "Loading xml files in $dir in progress \n";
opendir(BIN, $dir) or die "Can't open $dir: $!";
$commitCmd = '(cd ' . $commit_dir . '; ' . '/usr/bin/curl
http://localhost:7001/solr/update --data-binary @commit.txt -H
\'Content-type:text/plain; charset=utf-8\')';
while( defined ($file = readdir BIN) ) {
next if $file =~ /^\./;
$insertCmd = '(cd ' . $dir . '; ' . '/usr/bin/curl
http://localhost:7001/solr/update --data-binary @' . $file . ' -H
\'Content-type:text/plain; charset=utf-8\')';
system($insertCmd);
}
system($commitCmd);
closedir(BIN);
Thanks
Francis
-----Original Message-----
From: noble.p...@gmail.com [mailto:noble.p...@gmail.com] On Behalf Of Noble
Paul ??????? ??????
Sent: Thursday, July 09, 2009 10:40 PM
To: solr-user@lucene.apache.org
Subject: Re: Using curl comparing with using WebService::Solr
are these xml files in the solr add xml format?
When you post using curl, I guess it opens as many http connections as
there are files. if you can write a small program to post all these
files in one request, you should be able to get better perf.
the following can be the pseudo-code
open connection
write "<root>"
for each file
write filecontent
write "</root>"
close connection
On Fri, Jul 10, 2009 at 10:23 AM, Francis Yakin<fya...@liquid.com> wrote:
>
> I have about 1000 folders, each folder consist 2581 xml files. Total of xml
> files is ~ 2.6 millions
>
> I developed perl script, inside my script it's executed this cmd:
>
> curl http://localhost:7001/solr/update --data-binary "@0039000.xml" -H
> 'Content-type:text/plain; charset=utf-8'
>
> It tooks me about 4 1/2 hrs to load and commit.
>
> I would like to know the advantages using curl to posting/add/update the xml
> files to solr comparing with using WebService::Solr module?
>
> Is using WebService::Solr faster?
>
> The XML files are local on the Solr Master box, so I posting it locally( not
> using wan or lan).
>
> Any input will be much appreciated.
>
> Thanks
>
> Francis
>
>
>
--
-----------------------------------------------------
Noble Paul | Principal Engineer| AOL | http://aol.com
