Yes, you can "split" a large file into many small files. At the colo,
you can put them back together again. The command to put them back
together is "cat". The "join" command does something else, so you
don't want to try to use it.
NB: this is actually what the cat command is for. "cat" is short for
"concatenate". Although it's rarely used for this purpose! ;)
Alternatively, there are options for rsync that turn off the checksum
stuff. So if a file transfer gets interrupted, it picks off right
where it left off. You can then do file verification outside the
scope of rsync, e.g. by doing sha1sum on both sides and comparing the
results.
Contact your local sysadmins for assistance with either of these
options. :)
- Morty
On Wed, Apr 15, 2015 at 02:50:31PM -0400, Gang Fu wrote:
> We want to transfer the files to another location, 'colo' for disaster
> recovery. The long distance transfer is time-consuming and may fail
> sometimes.
>
> We are using rsync, and we believe rsync a 500 GB file or rsync many small
> files indeed make difference, since rsync does a checksum validation before
> transfer, so if a large portion of many small files have the same checksum,
> then we only need to transfer a small port of them.
>
> Can we just 'split' and 'join' db files before and after transferring?
>
> Best,
> Gang
>
> On Wed, Apr 15, 2015 at 1:17 PM, Morty <morty+virtu...@frakir.org> wrote:
>
> > On Tue, Apr 14, 2015 at 12:24:22PM -0400, Gang Fu wrote:
> >
> > > We want to copy a large virtuoso db from one server to another in
> > > different location. We cannot copy single 500 GB db file, which is
> > > slow and unstable. So we want to break the db files in different
> > > segments. I have tried with virtuoso striping: each segment has 20
> > > GB, and in total we have over 25 segments.
> >
> > What issue are you seeing with transferring a 500GB file?
> > Transferring one 500GB file should not be significantly slower than
> > transferring 25x 20GB files.
> >
> > If you are concerned about a transfer interruption, you could use
> > rsync. rsync has options to resume a failed transfer.
> >
> > Alternatively, you could use the Linux/Unix "split" command to split
> > the one large file into a bunch of smaller files.
> >
> > Or you could use the commercial version of virtuoso with built-in
> > replication.
> >
> > - Morty
> >
--
Mordechai T. Abzug
Linux red-sonja 3.11.0-24-generic #42-Ubuntu SMP Fri Jul 4 21:19:31 UTC 2014
x86_64 x86_64 x86_64 GNU/Linux
"A verbal contract isn't worth the paper it's written on." - Samuel Goldwyn
------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
Virtuoso-users mailing list
Virtuoso-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/virtuoso-users
