t; > > because they had talked about "replicating" the data.
> > > >
> > > > But we are not Pinterest, and do not expect to be taking in changes
> one
> > > > post at a time (at least the engineers don't - just wait until its
> us
Pinterest, and do not expect to be taking in changes one
> > > post at a time (at least the engineers don't - just wait until its used
> > for
> > > a Crud app that wants full-text search on a description field!).
> > Still,
> > > rsync can be very, very
one of them were using Solr replication,
> > > because they had talked about "replicating" the data.
> > >
> > > But we are not Pinterest, and do not expect to be taking in changes one
> > > post at a time (at least the engineers don't - just wait unti
ot expect to be taking in changes one
> > post at a time (at least the engineers don't - just wait until its used
> for
> > a Crud app that wants full-text search on a description field!).
> Still,
> > rsync can be very, very fast with the right options (-W for gigabit
&
est, and do not expect to be taking in changes one
> post at a time (at least the engineers don't - just wait until its used for
> a Crud app that wants full-text search on a description field!).Still,
> rsync can be very, very fast with the right options (-W for gigabit
> ethern
(at least the engineers don't - just wait until its used for
a Crud app that wants full-text search on a description field!). Still,
rsync can be very, very fast with the right options (-W for gigabit
ethernet, and maybe -S for sparse files). I've clocked it at 48 MB/s over
GigE prev
Hi Hoss,
Thanks for your reply, Please find answers to your questions below.
*Well, for starters -- have you considered at least looking into using the java
based Replicationhandler instead of the rsync scripts?*
- There was an attempt to to implement java based replication but it was
very slow
: However, we haven't yet implemented SolrCloud and still relying on
: distribution scripts - rsync, indexpuller mechanism.
Well, for starters -- have you considered at least looking into using hte
java based Replicationhandler instead of the rsync scripts?
Script based replication ha
ripts - rsync, indexpuller mechanism.
*Issue:*
I see that the indexes are getting created on indexer boxes, snapshots
being created and then pulled across to search boxes. The snapshots are
getting installed on search boxes as well. There are no errors in the
scripts logs and this process works
Sujatha,
A few thoughts:
* If you are still using rsync replication, you may be using an old version of
Solr. Consider upgrading (it's also more efficient with memory).
* Yes, rsync will "pollute" OS cache, but is it really pollution if it makes
the OS cache the index that is a
Hello,
When we migrated the solr webapps to a new server in production,We did an
rsync of all the indexes in our old server . On checking with Jconsole
,the OS free Ram was already filled to twice the Index size .There is no
other service in this server. The rsync was done twice.
Even though
rediffmail.com> wrote:
> >
> >>
> >> Hi,
> >>
> >> I am facing an issue while performing snapshot pulling thru Snappuller
> >> script from slave server :
> >> We have the setup of multicores on Master Solr and Slave Solr servers
ushar_kapoor...@rediffmail.com> wrote:
>
>>
>> Hi,
>>
>> I am facing an issue while performing snapshot pulling thru Snappuller
>> script from slave server :
>> We have the setup of multicores on Master Solr and Slave Solr servers.
>> Scenario , 2 cores
pshot pulling thru Snappuller
> script from slave server :
> We have the setup of multicores on Master Solr and Slave Solr servers.
> Scenario , 2 cores are set :
> i) CORE_WWW.ABCD.COM
> ii) CORE_WWW.XYZ.COM
>
> rsync-enable and rsync-start script run from CORE_WWW.ABCD.COM on
Hi,
I am facing an issue while performing snapshot pulling thru Snappuller
script from slave server :
We have the setup of multicores on Master Solr and Slave Solr servers.
Scenario , 2 cores are set :
i) CORE_WWW.ABCD.COM
ii) CORE_WWW.XYZ.COM
rsync-enable and rsync-start script run
On Fri, Mar 13, 2009 at 10:33 AM, sunnyfr wrote:
> And it was obvious how during snappuller on a slave server, the query time
> was a lot longer than the rest of the time.
Did the CPU utilization drop?
It could be writing of the new files being pulled forcing parts of the
current index files out
round 200msec with snappuller 3-6sec ..
Do you have any idea?
Thanks a lot,
--
View this message in context:
http://www.nabble.com/rsync-snappuller-slowdown-Qtime-tp22497625p22497625.html
Sent from the Solr - User mailing list archive at Nabble.com.
- Nutch
- Original Message
> From: revas
> To: solr-user@lucene.apache.org
> Sent: Friday, February 27, 2009 12:07:50 AM
> Subject: rsync
>
> When we rsync from master to slave ,since this refers to the same index that
> is on the master ,if master server goes down
p 22, 2008 at 6:49 AM, sunnyfr <[EMAIL PROTECTED]> wrote:
> >
> >>
> >> Yes but even if I run it I've no snapshot created, I don't get how I can
> >> fixe
> >> it.
> >>
> >>
> >>
> >>
> >> Bill Au w
snapshot created, I don't get how I can
>> fixe
>> it.
>>
>>
>>
>>
>> Bill Au wrote:
>> >
>> > You only need to run the rsync daemon on the master.
>> >
>> > Bill
>> >
>> > On Wed, Sep
>
> Bill Au wrote:
> >
> > You only need to run the rsync daemon on the master.
> >
> > Bill
> >
> > On Wed, Sep 17, 2008 at 10:54 AM, sunnyfr <[EMAIL PROTECTED]> wrote:
> >
> >>
> >> Hi Raghu,
> >>
> >> Thanks
Yes but even if I run it I've no snapshot created, I don't get how I can fixe
it.
Bill Au wrote:
>
> You only need to run the rsync daemon on the master.
>
> Bill
>
> On Wed, Sep 17, 2008 at 10:54 AM, sunnyfr <[EMAIL PROTECTED]> wrote:
>
>&g
You only need to run the rsync daemon on the master.
Bill
On Wed, Sep 17, 2008 at 10:54 AM, sunnyfr <[EMAIL PROTECTED]> wrote:
>
> Hi Raghu,
>
> Thanks it's clear now;
>
>
> Kashyap, Raghu wrote:
> >
> > Hi,
> >
> > Rsyncd is the rsync(
Hi Raghu,
Thanks it's clear now;
Kashyap, Raghu wrote:
>
> Hi,
>
> Rsyncd is the rsync(http://samba.anu.edu.au/rsync/) daemon. You need
> to make sure that Rsynchd is running on both master & the slave
> machines. You use snapshooter on the master server to
Hi,
Rsyncd is the rsync(http://samba.anu.edu.au/rsync/) daemon. You need
to make sure that Rsynchd is running on both master & the slave
machines. You use snapshooter on the master server to create the
snapshot & run snappuller on the slave machines to pull those snapshots
from maste
snappuller---rsync-tp19532941p19532941.html
Sent from the Solr - User mailing list archive at Nabble.com.
utch
- Original Message
From: Brian Whitman <[EMAIL PROTECTED]>
To: solr-user@lucene.apache.org
Sent: Wednesday, May 7, 2008 11:25:22 AM
Subject: force rsync update on a read-only index
We have a few slave solr servers that are just hardlinked-rsynced
copies of a master server
e.apache.org
> > Sent: Wednesday, May 7, 2008 11:25:22 AM
> > Subject: force rsync update on a read-only index
> >
> > We have a few slave solr servers that are just hardlinked-rsynced
> > copies of a master server.
> >
> > When we do the rsync the changes don
[EMAIL PROTECTED]>
> To: solr-user@lucene.apache.org
> Sent: Wednesday, May 7, 2008 11:25:22 AM
> Subject: force rsync update on a read-only index
>
> We have a few slave solr servers that are just hardlinked-rsynced
> copies of a master server.
>
> When we do the rsync the
We have a few slave solr servers that are just hardlinked-rsynced
copies of a master server.
When we do the rsync the changes don't show up immediately. The snap*
scripts call a commit on the slave servers -- but since these are
readonly servers we've disabled /update in the solr
: Be careful. 'rsync' has different meanings for 'directory' v.s.
: 'directory/'. I ran afoul of this.
eliminating some of the confusion about how rsync is one of the reasons
the rsyncd-start and snappuller scripts exist, and why a seperate rsyncd
port is
Be careful. 'rsync' has different meanings for 'directory' v.s.
'directory/'. I ran afoul of this.
-Original Message-
From: Walter Underwood [mailto:[EMAIL PROTECTED]
Sent: Wednesday, November 14, 2007 8:49 AM
To: solr-user@lucene.apache.org
Subject: Re: sn
: I'm not an rsync expert, but I beleive that /solr/ is a
: virtual directory defined in the rsyncd config. It is mapped
: to the real directory.
correct .. the real directory comes from the scripts.conf
:
: wunder
:
: On 11/14/07 8:43 AM, "Jae Joo" <[EMAIL PROTECTED]>
I'm not an rsync expert, but I beleive that /solr/ is a
virtual directory defined in the rsyncd config. It is mapped
to the real directory.
wunder
On 11/14/07 8:43 AM, "Jae Joo" <[EMAIL PROTECTED]> wrote:
> In the snappuller, the "solr" is hardcoded.
In the snappuller, the "solr" is hardcoded. Should it be
"${master_data_dir}?
# rsync over files that have changed
rsync -Wa${verbose}${compress} --delete ${sizeonly} \
${stats} rsync://${master_host}:${rsyncd_port}/solr/${name}/
${data_dir}/${name}-wip
Thanks,
Jae
: To completely remove the window of inconsistency, comment out the
: post-commit hook in solrconfig.xml that takes a snapshot, then send a
: commit to get a new snapshot and rsync from that.
i think yonik ment "UN-comment" the postCommit hook in the example
solrconfig.xml.
-Hoss
ery active, is
> constantly getting new docs and searched on. What can I do to
> preserve the index state while syncing?
To greatly reduce this window, take a "snapshot" of the index (do it
like snapshooter, or actually use that script) and rsync from that
snapshot instead.
To complet
Sep 28, 2007, at 5:41 PM, Yonik Seeley wrote:
It should... are there any errors in the logs? do you see the commit
in the logs?
Check the stats page to see info about when the current searcher was
last opened too.
ugh, nevermind.. was committing the wrong solr index... but Thanks
yonik fo
On 9/28/07, Brian Whitman <[EMAIL PROTECTED]> wrote:
> For some reason sending a
> is not refreshing the index
It should... are there any errors in the logs? do you see the commit
in the logs?
Check the stats page to see info about when the current searcher was
last opened too.
-Yonik
I'm not using snap* scripts but i quickly need to sync up two indexes
on two machines. I am rsyncing the data dirs from A to B, which work
fine. But how can I see the new index on B? For some reason sending a
is not refreshing the index, and I have to restart resin to
see it. Is there some
You are welcome.
Bill
On 9/21/07, Yu-Hui Jin <[EMAIL PROTECTED]> wrote:
>
> Bill,
>
> Thanks for the explanation. That helps my understanding on rsync and the
> replication in general.
>
>
> regards,
>
> -Hui
>
> On 9/20/07, Bill Au <[EMAIL PROTE
Bill,
Thanks for the explanation. That helps my understanding on rsync and the
replication in general.
regards,
-Hui
On 9/20/07, Bill Au <[EMAIL PROTECTED]> wrote:
>
> The "solr" that you are referring to in your third question in the
> name of the rsync area whic
The "solr" that you are referring to in your third question in the
name of the rsync area which is map to the solr data directory. This
is defined in the rsyncd configuration file which is generated on the
fly as Chris has pointed out. Take a look at rsyncd-start.
snappuller rsync
: So just to help my knowledge, where does this virtual setting of this "solr"
: string happen? Should it be in some config file or sth?
rsyncd-start creates an rsync config file on the fly ... much of it is
constants, but it fills in the rsync port using a variable from your
config.
-Hoss
r cases they usually use ${solr_root} to get to specific
> dirs.
> : ) I pasted this line again below:
>
> sorry ... i didn't realize you were talking about the script, i thought
> you were talking aboutthe docs.
>
> : I saw in snappuller line 226 (solr 1.2):
> :
> :
didn't realize you were talking about the script, i thought
you were talking aboutthe docs.
: I saw in snappuller line 226 (solr 1.2):
:
: ${stats} rsync://${master_host}:${rsyncd_port}/solr/${name}/
: ${data_dir}/${name}-wip
:
: Is the above "solr" a hard-coded solr home name? If s
${solr_root} to get to specific dirs.
) I pasted this line again below:
I saw in snappuller line 226 (solr 1.2):
${stats} rsync://${master_host}:${rsyncd_port}/solr/${name}/
${data_dir}/${name}-wip
Is the above "solr" a hard-coded solr home name? If so, it's not desirable
sin
: 1) config different port for each solr home dir (since they run on the same
: host);
you mean a differnet rsync port right? ... yes the scripts as distributed
assume that each rsync daemon will be dedicated to a single solr
"instance" .. the idea beaing that even if you have 12 Sol
Ok, I should correct myself. For #1, I think we need to
1) config different port for each solr home dir (since they run on the same
host);
2) run rsync-start script under each of the solr home's bin dir.
(btw, just to make clear, we should run rsync-start after rsync-enable that
I under
Hi, there,
So we are using the Tomcat's JNDI method to set up multiple solr instances
within a tomcat server. Each instance has a solr home directory.
Now we want to set up collection distribution for all these solr home
indexes. My understanding is:
1. we only need to run rsync-start onc
reloading would
> make this more difficult.
rsync will only copy over the changed index files, not the whole index
each time.
-Yonik
: Bill Au <[EMAIL PROTECTED]>
To: solr-user@lucene.apache.org; jason rutherglen <[EMAIL PROTECTED]>
Sent: Wednesday, March 29, 2006 6:07:35 AM
Subject: Re: Rsync
The segments file will change when new segments are created. But what I
really
meant before was that the file deletable also
slave
>
> Lucene can certainly record a list of all new segment files added. I
> think the tricky part
> is to ensure that a consistent copy of the index is being distributed.
>
> Bill
>
>
> On 3/27/06, jason rutherglen <[EMAIL PROTECTED]> wrote:
> >
> >
ble to avoid using rsync and record a
> list of all new segment files added (from within Lucene), and simply use
> HTTP to sync down the newest ones? Perhaps only using rsync after an
> optimize? Seems like if I understand Lucene correctly only new files are
> created?
>
>
>
I was thinking, would it not be possible to avoid using rsync and record a list
of all new segment files added (from within Lucene), and simply use HTTP to
sync down the newest ones? Perhaps only using rsync after an optimize? Seems
like if I understand Lucene correctly only new files are
55 matches
Mail list logo
