Hi,
Is the new replication feature based on HTTP requests between sites ?
If yes, then I guess it might be possible to configure an HTTP server
with mod_deflate so the data is compressed on the fly.
C.
Simon Collins wrote:
I have now optimized the index - down to 325mb, it compresses down to 20mb.
I think the new replication thing in solr is great, but if it could compress
the files it's sending, it would be an awful lot more useful when replicating,
as we are, between sites.
--------------------------------------------------------
Simon Collins
Systems Analyst
Telephone: 01904 606 867
Fax Number: 01904 528 791
shoe-shop.com ltd
Catherine House
Northminster Business Park
Upper Poppleton, YORK
YO26 6QU
www.shoe-shop.com
--------------------------------------------------------
This message (and any associated files) is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential, subject to copyright or constitutes a trade secret. If you are not the intended recipient you are hereby notified that any dissemination, copying or distribution of this message, or files associated with this message, is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer. Messages sent to and from us may be monitored.
Internet communications cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. Therefore, we do not accept responsibility for any errors or omissions that are present in this message, or any attachment, that have arisen as a result of e-mail transmission. If verification is required, please request a hard-copy version. Any views or opinions presented are solely those of the author and do not necessarily represent those of the company. (PAVD001)
Shoe-shop.com Limited is a company registered in England and Wales with company number 03817232. Vat Registration GB 734 256 241. Registered Office Catherine House, Northminster Business Park, Upper Poppleton, YORK, YO26 6QU.
-----Original Message-----
From: Noble Paul നോബിള് नोब्ळ् [mailto:[EMAIL PROTECTED]
Sent: 29 October 2008 03:29
To: solr-user@lucene.apache.org
Subject: Re: replication handler - compression
The new replication feature does not use any unix commands , it is
pure java. On the fly compression is hard but possible.
I wish to repeat the question. Did you optimize the index? Because a
10:1 compression is not usually observed in an optimized index. Our
own experiments showed compression of around 10:6 for optimized
indexes.
--Noble
On Wed, Oct 29, 2008 at 3:41 AM, Lance Norskog <[EMAIL PROTECTED]> wrote:
Aha! The hint to the actual problem: "When compressed with winzip". You are
running Solr on Windows.
Snapshots don't work on Windows: they depend on a Unix file system feature. You
may be copying the entire index. Not just that, it could be inconsistent.
This is a fine topic for a "best practices for Windows" wiki page.
The 'scp' program what you want. It has an option to compress on the fly
without saving anything to disk. 'Rcopy' in particular has features to only
copy what is not already at the target. The Putty suite 'pscp' program also
has the compression feature.
Lance
-----Original Message-----
From: Noble Paul നോബിള് नोब्ळ् [mailto:[EMAIL PROTECTED]
Sent: Monday, October 27, 2008 9:36 PM
To: solr-user@lucene.apache.org
Subject: Re: replication handler - compression
It is useful only if your bandwidth is very low.
Otherwise the cost of copying/comprressing/decompressing can take up
more time than we save.
I mean compressing and transferring. If the optimized index itself has a very
high compression ratio then it is worth exploring the option of compresssing
and transferring. And do not assume that all the files in the index directory
is transferred during replication. It only transfers the files which are used
by the current commit point and the ones which are absent in the slave
On Tue, Oct 28, 2008 at 2:49 AM, Simon Collins
<[EMAIL PROTECTED]> wrote:
Is there an option on the replication handler to compress the files?
I'm trying to replicate off site, and seem to have accumulated about
1.4gb. When compressed with winzip of all things i can get this down
to about 10% of the size.
Is compression in the pipeline / can it be if not!
simon
This message has been scanned for malware by SurfControl plc.
www.surfcontrol.com
--
--Noble Paul
--
--Noble Paul
