Subversion for object code
Our software products use different components, where many base components are used in different products. It is not necessary that all developpers compile these componentens themselves, that is, these components are distributed binary (as object files or libraries). To make the distribution of binaries easier, some developers started to include also object files and libraries of the common base components in the subversion repository. I know that subversion was not invented for that purpose, but from our experience it seems that it actually works. My questions: What kind of problems could occur if too many binaries are in the subversion archive? How good does subversion make diffs of object code? What better options for sharing versions of object fils are available? Helmut -- NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone! Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
Re: Subversion for object code
> Datum: Tue, 28 Feb 2012 11:47:43 +0100
> Von: Stephen Butler
Thank you for the quick answer.
> But it also has disadvantages:
>
> - Runaway repository growth. Object files and .jar files don't compress
> as well as text. If you bust a hard limit for your repository disk space,
> your IT service provider might force you to pay a drastic penalty.
This is what I am afraid of; see also below ("representation-sharing")
> - Slower checkouts, updates, and merges due to working copy size
To prevent this, we can make a suitable seperation of binaries and sources in
the directory tree.
> Also, you'll miss the features of language-specific dep-mgt tools, which
> have a lot of sanity checks built in. A few examples:
>
> Java: Maven
> Python: virtualenv + pip
> Ruby: bundler + gem
We use C++.
> > How good does subversion make diffs of object code?
>
> By default, 'svn diff' skips binary files. You can customize it to use
> another
> program to display diffs for, say, "*.o" file.
This we do not need.
> There's a quick summary of binary-file handling here:
>
> http://svnbook.red-bean.com/en/1.7/svn.forcvs.binary-and-trans.html
>
> On the server side, Subversion stores files using a binary diff algorithm,
> and has a "representation-sharing" feature for avoiding redundant data
> storage.
Acutually this "representation-sharing" was my question. How good does it work
for compiled C++ code? How much does the repository typically grow?
> > What better options for sharing versions of object fils are available?
>
> That depends on your programming language.
C++
Helmut
--
NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone!
Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
Re: Subversion for object code
> Datum: Tue, 28 Feb 2012 14:06:14 +0100 > Von: Ulrich Eckhardt > An: users@subversion.apache.org > Betreff: Re: Subversion for object code > > I know that subversion was not invented for that purpose > > I tend to disagree a bit, Subversion was intended to version non-textual > resources, too, and you are not doing something that is unsupported and > discouraged and left without support either. > > > > but from our experience it seems that it actually works. > > Yes, I'd even call it common use, even though it's not the majority. Quite interesting, these are good news. > All files are binaries. The only issue here is the size they take, > because every change requires some storage in the repository. Did someone already made measurements how big the additional storage consumption is for incremental commits of object files, e.g. created from C++ source? > You probably already know the typical trunk/branches/tags hierarchy. > Keep this hierarchy, but also keep it clean of object files, because > merging and diffing them just doesn't work. Instead, add a fourth folder > with release packages (e.g. bin). OK, this we observe already (Actuallay we do not use fourth top-folder, but we have a clear seperation of source and bin at some other level in the hierarchy) > BTW: Another approach is to use a build server that e.g. runs nightly > and stores the results on a network share. This does not work in our situation because for some components we do not want to use the latest but some stable older version. Helmut -- NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone! Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
Re: Subversion for object code
> Datum: Tue, 28 Feb 2012 13:43:05 +0100 > Von: Stephen Butler > > Acutually this "representation-sharing" was my question. How good does > it work for compiled C++ code? How much does the repository typically grow? > It's best to write a simple script that loads various versions of your > own files into an empty repository. Using the default FSFS backend, > you can see the size of each commit. OK, I now made some experiments with a typical library (2 MB object code, 2MB library size) comiled with Visual C++. Just completely recompiling the library (unchanged source) and commitiong again gives a repository growth of 7 KB (0,15% of the "commit size") If I add one member function (to approx 40), which means that I also changed the header and had to recompile some other classes, and commit, the repository grows by 130 KB (3% of "commit size"). My interpretation of this is that "representation-sharing" works well enough for compiled C++ code, i.e. if I change 2% of source code and commit the changed objects and libs, then typically a comparable amount (+/- 2%) of the object code is additionally added to the repostory and the remaining +/- 98% can be reused from previously committed object code. Helmut -- NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone! Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
Re: Subversion for object code
Original-Nachricht > Datum: Thu, 1 Mar 2012 09:07:41 +0200 > Von: Daniel Shahaf > Representation sharing works only on complete files. (If two files are > not byte-for-byte identical, it never kicks in.) What you see would be > the xdelta binary-diff algorithm being efficient. OK. Does this mean that svn uses internally the same algorithm as xdelta (http://code.google.com/p/xdelta/) This would mean that I could directly use xdelta3 -s SOURCE TARGET > OUT to estimate how big the differences between binary files are (without creating some svn test repository and measuring repostiory size) Helmut -- NEU: FreePhone 3-fach-Flat mit kostenlosem Smartphone! Jetzt informieren: http://mobile.1und1.de/?ac=OM.PW.PW003K20328T7073a
