I was thinking, would it not be possible to avoid using rsync and record a list of all new segment files added (from within Lucene), and simply use HTTP to sync down the newest ones? Perhaps only using rsync after an optimize? Seems like if I understand Lucene correctly only new files are created?
