Re: Repository organization for complex project

[Stephen Connolly]

On 13 October 2010 19:19, David Weintraub  wrote:
> On Wed, Oct 13, 2010 at 1:42 PM, Les Mikesell  wrote:
>> How would you access them if you don't have java/maven/ivy when you want to
>> retrieve a certain version?
>
> Before we adopted our Ant projects to use Maven, we simply used the
>  task. You can always download from a Maven repository using the
> "get" and "wget" commands. Maven repositories, after all, are HTTP
> accessible. There's no real reason why you can't user wget and curl to
> do the fetching if you aren't a Java shop.
>
> Uploading is trickier though. We simply used a shell script that
> generated the correct "mvn install-file" command and didn't have to
> generate a POM. Maven and Java are fairly easy to install and use open
> source licenses, so there isn't really a big deal to install them.
> With the "mvn install-file", we didnt' even have to create a pom.xml
> file. We could build the command on the fly with just a shell script.

Our C/C++ guys just use curl to POST the binaries to Nexus over
http... we also POST the .pom file and the .md5 and .sha1 files...
that is because one of their build toolchain envs cannot have Java on
it... Nexus will rebuild the metadata.xml files for you, so all you
really need is to post the .pom and the e.g. .so and the .pom.md5,
.pom.sha1, .so.md5 and .so.sha1 files and you're done.

If you have a java install on the system, use maven or maven-ant-tasks
(from ant) or ivy (from ant) to do the post for you.

I think I had a set of bash scripts somewhere in
http://svn.apache.org/repos/asf/maven/sandbox/trunk/ at some time.
I'll see if I can find them

>
> But, there's really no reason that you HAVE to use a Maven repository.
> All you need is a remote storage directory and the concept of version
> numbers. I'd setup the repository to use Mavenish concept of object
> with sub-directories for each release and embedd the release into the
> object between the object's name and suffix. For example, if Project B
> produced a library libprojb.so, I'd call the thing libprojb-3.2.so,
> and store it in my projectb/3.2 directory.
>

The advantage of a Maven repository is that you can use a Maven
Repository Manager to enforce some good things, such as write-once,
authorization and search... but there is nothing stopping you from
just copying everything to a big file share in the sky!

> Imagine if you take the approach of checking into your repository the
> libprojb.so library for Project A to use. What usually happens is that
> you have a lib directory under your project with a libprojb.so
> directory in it. You have no idea what files went into this directory
> or what version it was from. Two years from now, that libprojb.so
> directory will still be there and no one will know anything about it.
>
> However, imagine if in Project A's make file is a macro
> PROJB-LIB=projb/3.2, and later on, it uses this information to do a
> scp from the release repository. You can look at that particular
> version of Project A and say "Ah! It uses version 3.2 of Project B.
> Since we tagged the source code used for that version of Project B as
> 3.2, we can still trace down the correct version. And, we can see that
> the next release of Project A uses version 3.3 of Project B.
>
> --
> David Weintraub
> qazw...@gmail.com
>

Re: Repository organization for complex project

[Stephen Connolly]

On 13 October 2010 21:42, Les Mikesell  wrote:
> On 10/13/2010 2:52 PM, BRM wrote:
>>
>>> From: Les Mikesell
>>> We currently commit component binaries back into  tags which are then
>>> used by
>>> other things with external references, and final  binaries are managed
>>> separately by project, but that seems wrong on several  levels.
>>
>> For the projects we do nightly builds on we use an FTP site that the build
>> scripts upload the builds to. This is easy enough to script even on
>> Windows -
>> whether using DOS Batch or Windows Scripting Host.
>> I'm sure Hudson and other build systems can handle that just fine.
>>
>>> I'd like to find a generic scheme (and one that can be plugged  into
>>> hudson if
>>> possible) to store build results with a versioning scheme that
>>>
>>> doesn't force you to keep them forever because most are obsolete after a
>>> few
>>> new  builds but that has a close-coupling to the svn version
>>> or at least the tag name  of the matching source.  But this seems like it
>>> should be a common scenario  and not something everyone re-invents
>>> differently.
>>
>> Well, right now I have a lot of projects that don't have a nightly build -
>> mostly due to how Qt and Visual Studios interact - or rather, the problems
>> in
>> doing so.
>> However, you can do a couple things to track this very well and
>> automatically,
>> partially supported via Subversion's Keyword Substitution.
>> I use $HeadUrl$ in a number of projects to get the URL, which I then parse
>> to
>> get a information about the build; for example, if it's a tag then I grab
>> the
>> release information, or a branch then I build a special branch info to
>> output.
>>
>> For things like the global revision number you'll need to use 'svnversion'
>> to
>> capture the output somehow.
>>
>> The following would probably work for you:
>>
>> 1. Capture the URL somewhere as part of checking it out using $HeadURL$
>> 2. As part of the build script run and capture the output of "svnversion",
>> you
>> might want to do so with "svn info" too - though this only works if you
>> are NOT
>> doing an export.
>>
>> You can then store these with a copy of the source and resulting binaries
>> to
>> have full traceability.
>> Alternatively, if you are using "svn co" in the build, just tarball or zip
>> up
>> the project's sandbox at the end of the build script and save it as a
>> whole -
>> this will keep ALL the information, URLs, versions, etc; for you - and is
>> very
>> build system friendly too.
>
> This covers the 'backtrack from binary' scenario as long as you keep those
> extra pieces in the zip with them.  But I'm looking more for a filesystem
> and/or http URL name/path mapping to store the binaries so you can easily
> find the currently relevant versions in the first place.

Use Nexus (A maven repository manager)

The backing store is on the file system.

It has a searchable interface that can be used over http to, e.g. get
the latest version (IIRC even the latest version within a range, e.g
[1.0,2.0) will ensure I never get any of the 2.0 artifacts)

It's very lightweight and almost trivial to install.

Oh and the open source edition is free! I've used the pro edition and
it's great but the OSS is fine for most use cases... the only thing I
miss from the pro is the staging support

-Stephen

P.S. I am just a happy Nexus user, I have no affiliation with Sonatype
at present.

> Among other things
> it should provide a straightforward way to replace our current svn externals
> to pull specific libraries into place for dependent builds, and also map to
> our branch/tag name structure for qa and production releases.  Maven may be
> the right answer, but I was thinking in terms of something that could be
> viewed as a filesystem - maybe like alfresco where additional rules can be
> imposed.
>
> --
>  Les Mikesell
>    lesmikes...@gmail.com
>
>

RE: Subversion on AIX

[Giulio Troccoli]

>


Linedata Limited
Registered Office: 85 Gracechurch St., London, EC3V 0AA
Registered in England and Wales No 3475006 VAT Reg No 710 3140 03

-Original Message-


> From: David Weintraub [mailto:qazw...@gmail.com]
> Sent: 13 October 2010 18:03
> To: Giulio Troccoli
> Cc: Subversion
> Subject: Re: Subversion on AIX
>
> I was able to build everything until neon. There I get
>
>$ ./configure --with-expat=/app/fms/build/lib/libexpat.la
> --enable-shared=yes --prefix=/app/fms/build
> checking for a BSD-compatible install... ./install-sh -c
> checking for gcc... gcc
> checking for C compiler default output file name...
> configure: error: in `/app/fms/build/subversion-1.6.13/neon':
> configure: error: C compiler cannot create executables
> See `config.log' for more details.
> cmu...@fmsdwbap01:~/subversion-1.6.13/neon
> $
>
> Can you give me any help? It looks like it's crapping out
> when it is trying to determine the default link output.
> However, apr, apr-util, and expat all worked.
>

I didn't use gcc but cc

CC="/usr/vac/bin/cc" \
CFLAGS="-qmaxmem=-1 -O2 -qlanglvl=extended" \
CPPFLAGS="-I/usr/local/include" \
LDFLAGS="-brtl" \
./configure \
--with-expat=/usr/local/lib/libexpat.la \
--enable-shared=yes

Can you try that? Adjust paths for expat according to your previous build

G

UTF-16 files and inconsistent line endings

[David Aldrich]

Hi

I am developing with Visual C++ Express 2008 on Windows. I needed to add some 
Japanese characters to a source file, whereupon the editor stored the file in 
UTF-16LE encoding. Subversion (1.6.12) now complains that the file has 
inconsistent EOL-style.

Does Subversion 1.6 support UTF-16LE encoding?

Is there a workaround for this, such as setting eol-style to CRLF (i.e. not 
native)? 

Best regards 

David 

Re: UTF-16 files and inconsistent line endings

[Stefan Sperling]

On Thu, Oct 14, 2010 at 09:29:50AM +0100, David Aldrich wrote:
> Hi
> 
> I am developing with Visual C++ Express 2008 on Windows. I needed to add some 
> Japanese characters to a source file, whereupon the editor stored the file in 
> UTF-16LE encoding. Subversion (1.6.12) now complains that the file has 
> inconsistent EOL-style.
> 
> Does Subversion 1.6 support UTF-16LE encoding?

Only if you mark the file as binary.

> Is there a workaround for this, such as setting eol-style to CRLF (i.e. not 
> native)? 

Mark the file as binary.
See http://subversion.tigris.org/issues/show_bug.cgi?id=2194

Stefan

Re: Bug report against SVN 1.6.13

[Stefan Sperling]

On Thu, Oct 14, 2010 at 01:42:36AM +0200, Paul Maier wrote:
> Hi there! 
>  
> file b should be read-write here; what do you think: 
>  
>  
> Reproduction script: 
>  
> svn --version 
> svnadmin create xx 
> svn co "file:///C:/[...]/xx" yy 
> cd yy 
> echo a > a 
> svn add a 
> svn propset svn:needs-lock "*" a 
> svn ci -m "" 
> svn up 
> svn cp a b 
> ls -lA 
>  
>  
> Observed behaviour: 
> --- 
> File b is read-only. 
> No svn command is available to make file b read-write. 
> svn lock won't do, because file b is not in the repository yet. 
> Even a "svn lock a" before the copy doesn't help out. 
>  
>  
> Expected behaviour: 
> --- 
> File b is read-write. 
> Files without representation in the repository (files that come from a
> uncommitted svn cp or svn mv) should be read-write regardless
> svn:needs-lock setting. 

I'm fine with this idea if the file is also made read-only after commit.

Would you like to try to write a patch against Subversion trunk that
implements this feature? (I wouldn't really call it a "bug". A bug is
when something doesn't work as designed. In this case, it seems like
the designers of the lock feature didn't care or overlooked this issue,
so the desired behaviour is simply "unspecified".)

Stefan

Re: UTF-16 files and inconsistent line endings

[Ulrich Eckhardt]

On Thursday 14 October 2010, David Aldrich wrote:
> I am developing with Visual C++ Express 2008 on Windows. I needed to add
> some Japanese characters to a source file, whereupon the editor stored the
> file in UTF-16LE encoding.

Subversion doesn't support UTF-16 as text file, it treats it as unknown (raw 
binary) file...

> Subversion (1.6.12) now complains that the file has inconsistent EOL-style.

...and that doesn't mix well with this text-specific setting.

You could try using UTF-8 instead, though I'm not sure how to explain that to 
MSVC, maybe just storing with a BOM is enough. MSVC allows saving as UTF-8 
when choosing "save as" in the "save" button's drop-down list, notepad.exe 
does that if you save as UTF-8, IIRC.


> Is there a workaround for this, such as setting eol-style to CRLF (i.e. not
> native)?

Simply deleting the svn:eol-style property would also silence the warning.

Uli


-- 
ML: http://subversion.apache.org/docs/community-guide/mailing-lists.html
FAQ: http://subversion.apache.org/faq.html
Docs: http://svnbook.red-bean.com/

Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932

**
Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland
Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
**
   Visit our website at 
**
Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten 
bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen 
Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein 
sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, 
weitergeleitet, veröffentlicht oder anderweitig benutzt werden.
E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte 
Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht 
verantwortlich.
**

RE: UTF-16 files and inconsistent line endings

[David Aldrich]

Hi Ulrich and Stefan

Thanks for your replies. Using UTF-8 has solved the problem, as you suggested.

Best regards 

David 


> -Original Message-
> From: Ulrich Eckhardt [mailto:eckha...@satorlaser.com]
> Sent: 14 October 2010 12:23
> To: users@subversion.apache.org
> Subject: Re: UTF-16 files and inconsistent line endings
> 
> On Thursday 14 October 2010, David Aldrich wrote:
> > I am developing with Visual C++ Express 2008 on Windows. I needed to add
> > some Japanese characters to a source file, whereupon the editor stored the
> > file in UTF-16LE encoding.
> 
> Subversion doesn't support UTF-16 as text file, it treats it as unknown (raw
> binary) file...
> 
> > Subversion (1.6.12) now complains that the file has inconsistent EOL-style.
> 
> ...and that doesn't mix well with this text-specific setting.
> 
> You could try using UTF-8 instead, though I'm not sure how to explain that to
> MSVC, maybe just storing with a BOM is enough. MSVC allows saving as UTF-8
> when choosing "save as" in the "save" button's drop-down list, notepad.exe
> does that if you save as UTF-8, IIRC.
> 
> 
> > Is there a workaround for this, such as setting eol-style to CRLF (i.e. not
> > native)?
> 
> Simply deleting the svn:eol-style property would also silence the warning.
> 
> Uli
> 
> 
> --
> ML: http://subversion.apache.org/docs/community-guide/mailing-lists.html
> FAQ: http://subversion.apache.org/faq.html
> Docs: http://svnbook.red-bean.com/
> 
> Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland
> Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
> 
> *
> *
> Sator Laser GmbH, Fangdieckstraße 75a, 22547 Hamburg, Deutschland
> Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932
> *
> *
>Visit our website at 
> *
> *
> Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten
> bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen
> Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein
> sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen,
> weitergeleitet, veröffentlicht oder anderweitig benutzt werden.
> E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte
> Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht
> verantwortlich.
> *
> *
> 
> 
> 
>  Click
> https://www.mailcontrol.com/sr/j7TZ4!efwiHTndxI!oX7Ug3O71j94L3XM4taKVk7xLgF5u
> 8fC7h07Llmcw5pzXf0QETRMEibH6EpdY6WjzZSeg==  to report this email as spam.

Re: Subversion on AIX

[David Weintraub]

Alas, we don't have "cc", only "gcc".

I finally got everything to work. I had to fudge the flags a bit. And,
I discovered that it wants a later version of Neon than came with the
Subversion package. (I believe the depedencies came with .28 and
Subversion said it needed .29 as a minimum.)

I had to manually compile zlib and set the --with-zlib= flag. It took
me a while to figure out that this flag isn't suppose to point to the
directory where libz.a exists, but instead points to the directory
that contains both the includes and lib directory where the zlib
headers and library are located.

APR presented another nasty trick. When I compiled APR, it created a
liba*.la files that are actually texts file and contains path
information. Despite compiling APR with the path /home/cmuser/lib, and
the fact that it actually installed there, it had set libdir to
/usr/local/apr/lib. I had to manually change all of these.

So, I now have 1.6.13 working on my AIX 5.3 system, but the http
protocol won't work. If I want that to work, I'll have to download the
latest copy of neon from the Apache neon project and recompile
everything again. Fortunately, we're going to try using svnserve, so
that may not be an issue.

I may have already stated this, but the way the build works really
sucks. Running "configure" on AIX powerpc would take 15 to 30 minutes.
Each time, determining whether my version of sed truncates characters,
the name of my default executable created by the linker, etc. And,
when things didn't work, it took hours to trace where the error was
occurring and why. The "configure" shell script is a very long and
confusing mess with almost no documentation.

I do have my build commands in a shell script in case anyone has to do
this again (I did this when I realized I had to do this over and over
again, and the entire Subversion build takes almost 3 hours to do).

I may try again with the latest Neon if we do need the http protocol.

On Thu, Oct 14, 2010 at 3:55 AM, Giulio Troccoli
 wrote:
>>
>
>
> Linedata Limited
> Registered Office: 85 Gracechurch St., London, EC3V 0AA
> Registered in England and Wales No 3475006 VAT Reg No 710 3140 03
>
> -Original Message-
>
>
>> From: David Weintraub [mailto:qazw...@gmail.com]
>> Sent: 13 October 2010 18:03
>> To: Giulio Troccoli
>> Cc: Subversion
>> Subject: Re: Subversion on AIX
>>
>> I was able to build everything until neon. There I get
>>
>>    $ ./configure --with-expat=/app/fms/build/lib/libexpat.la
>> --enable-shared=yes --prefix=/app/fms/build
>>     checking for a BSD-compatible install... ./install-sh -c
>>     checking for gcc... gcc
>>     checking for C compiler default output file name...
>>     configure: error: in `/app/fms/build/subversion-1.6.13/neon':
>>     configure: error: C compiler cannot create executables
>>     See `config.log' for more details.
>>     cmu...@fmsdwbap01:~/subversion-1.6.13/neon
>>     $
>>
>> Can you give me any help? It looks like it's crapping out
>> when it is trying to determine the default link output.
>> However, apr, apr-util, and expat all worked.
>>
>
> I didn't use gcc but cc
>
> CC="/usr/vac/bin/cc" \
> CFLAGS="-qmaxmem=-1 -O2 -qlanglvl=extended" \
> CPPFLAGS="-I/usr/local/include" \
> LDFLAGS="-brtl" \
> ./configure \
> --with-expat=/usr/local/lib/libexpat.la \
> --enable-shared=yes
>
> Can you try that? Adjust paths for expat according to your previous build
>
> G
>



-- 
David Weintraub
qazw...@gmail.com

Re: Repository organization for complex project

[David Weintraub]

On Thu, Oct 14, 2010 at 3:39 AM, Stephen Connolly
 wrote:
> Our C/C++ guys just use curl to POST the binaries to Nexus over
> http... we also POST the .pom file and the .md5 and .sha1 files...
> that is because one of their build toolchain envs cannot have Java on
> it... Nexus will rebuild the metadata.xml files for you, so all you
> really need is to post the .pom and the e.g. .so and the .pom.md5,
> .pom.sha1, .so.md5 and .so.sha1 files and you're done.

I would think that you'd need whatever protocol Maven uses to actually
put files on the repository. I guess the Maven protocol is simpler
than I thought and simply uses webdav authentication and the mvn
file::deploy simply calculates the URL for you.

That's good to know. That means you can replace all of Maven with some
fairly simple shell scripts using curl. Then in a non-Java project,
all the Maven repositories do is provide a nice interface for
searching and administration of a HTTP based repository.

-- 
David Weintraub
qazw...@gmail.com

RE: Subversion on AIX

[Loritsch, Berin]

> I may have already stated this, but the way the build works 
> really sucks. Running "configure" on AIX powerpc would take 
> 15 to 30 minutes.
> Each time, determining whether my version of sed truncates 
> characters, the name of my default executable created by the 
> linker, etc. And, when things didn't work, it took hours to 
> trace where the error was occurring and why. The "configure" 
> shell script is a very long and confusing mess with almost no 
> documentation.

The configure script is autogenerated from a couple other files.  I
believe it is using the standard GNU autoconf stack to generate the
config and make files.  The original files are M4 scripts (even uglier
to look at).  Unfortunately, to make the build easier on AIX requires
someone with AIX experience working with the GNU autoconf/automake
teams.  I get the feeling that's hard to come by, which would explain
why it's such a pain.

On Linux (and probably BSD), the same process is much quicker and is not
filled with such nasty problems.  In my brief encounters with Solaris, I
know there are some big enough differences in the platforms to cause
plenty of heartburn.  I've done very little with AIX, but it just feels
different from both Solarix and Linux or BSD.  I shudder to think of the
experience using Irix would be like building this...


Confidentiality Note: The information contained in this message, and any 
attachments, may contain proprietary and/or privileged material. It is intended 
solely for the person or entity to which it is addressed. Any review, 
retransmission, dissemination, or taking of any action in reliance upon this 
information by persons or entities other than the intended recipient is 
prohibited. If you received this in error, please contact the sender and delete 
the material from any computer.


Confidentiality Note: The information contained in this message, and any 
attachments, may contain proprietary and/or privileged material. It is intended 
solely for the person or entity to which it is addressed. Any review, 
retransmission, dissemination, or taking of any action in reliance upon this 
information by persons or entities other than the intended recipient is 
prohibited. If you received this in error, please contact the sender and delete 
the material from any computer.

sparse checkout tool

[Jeremy Mordkoff]

I remember reading about a tool that let you define what parts of a
repository were needed to create a work space. It basically did a bunch
of sparse checkouts based on a config file. 

Could someone remind me what was its name? Please reply directly as I'm
no longer on the list.

Thanks
JLM

Re: sparse checkout tool

[Mark Phippard]

On Thu, Oct 14, 2010 at 1:38 PM, Jeremy Mordkoff  wrote:
> I remember reading about a tool that let you define what parts of a
> repository were needed to create a work space. It basically did a bunch
> of sparse checkouts based on a config file.
>
> Could someone remind me what was its name? Please reply directly as I'm
> no longer on the list.

You must mean this:

http://svn.apache.org/viewvc/subversion/trunk/tools/client-side/svn-viewspec.py?view=markup



-- 
Thanks

Mark Phippard
http://markphip.blogspot.com/

Re: svn Farm

[David Brodbeck]

On Sat, Oct 9, 2010 at 6:39 AM, Nico Kadel-Garcia  wrote:
> Second, many working environments in the UNIX world rely on NFS based
> home directoies, to share working environments and configurations
> across a variety of machines. In such environments, *any* host that
> can be leveraged to local root access can "su" or "suco" to become the
> target user, and access their entire home directory.
>
> Think I'm kidding? Walk into any university environment: plug in a
> live Linux CD. Run an "nmap" scan for hosts running NFS. Run
> "showmount" to detect what NFS shares are published to everyone. Go
> ahead and mount the shares. Look in them for home directoriies. Look
> in them, using your local root privileges, for Subversion passphrases.
> (Look for CVS passphrases and un-passphrase-protected SSH keys while
> you're at it.)

This is why running public-facing NFS servers using auth_sys and
no_root_squash is a BAD idea.  If this is happening at your site, you
have much more serious things to worry about than subversion passwords
being stolen.  For example, in your scenario it would be trivial to
create an suid-root shell binary, which a local user could then run
and gain root privileges.

-- 
David Brodbeck
System Administrator, Linguistics
University of Washington

Re: svn Farm

[Stefan Sperling]

On Thu, Oct 14, 2010 at 10:56:35AM -0700, David Brodbeck wrote:
> On Sat, Oct 9, 2010 at 6:39 AM, Nico Kadel-Garcia  wrote:
> > Second, many working environments in the UNIX world rely on NFS based
> > home directoies, to share working environments and configurations
> > across a variety of machines. In such environments, *any* host that
> > can be leveraged to local root access can "su" or "suco" to become the
> > target user, and access their entire home directory.
> >
> > Think I'm kidding? Walk into any university environment: plug in a
> > live Linux CD. Run an "nmap" scan for hosts running NFS. Run
> > "showmount" to detect what NFS shares are published to everyone. Go
> > ahead and mount the shares. Look in them for home directoriies. Look
> > in them, using your local root privileges, for Subversion passphrases.
> > (Look for CVS passphrases and un-passphrase-protected SSH keys while
> > you're at it.)
> 
> This is why running public-facing NFS servers using auth_sys and
> no_root_squash is a BAD idea.  If this is happening at your site, you
> have much more serious things to worry about than subversion passwords
> being stolen.  For example, in your scenario it would be trivial to
> create an suid-root shell binary, which a local user could then run
> and gain root privileges.

Exactly. Bad NFS configuration isn't Subversion's fault.
Neither are NFS implementations that have insecure default settings,
like not mapping 'root' to 'nobody' by default.

There are problems with plaintext passwords, no doubt.
But the above scenario description is hyped up and misses the point.
If you cannot trust root on a UNIX box, don't save anything of value
on that box.

As of 1.6, Subversion asks the user before saving passwords in
plaintext. 1.6 also added support for using GNOME Keyring and KDE Wallet
as password stores.

It looks like there will be support for using PGP to encrypt passwords soon.
Maybe even in 1.7. Some code for this has already entered the repository:
http://svn.apache.org/viewvc?view=revision&revision=1005036

Stefan

Re: Repository organization for complex project

[Les Mikesell]

On 10/14/2010 8:24 AM, David Weintraub wrote:

On Thu, Oct 14, 2010 at 3:39 AM, Stephen Connolly
  wrote:

Our C/C++ guys just use curl to POST the binaries to Nexus over
http... we also POST the .pom file and the .md5 and .sha1 files...
that is because one of their build toolchain envs cannot have Java on
it... Nexus will rebuild the metadata.xml files for you, so all you
really need is to post the .pom and the e.g. .so and the .pom.md5,
.pom.sha1, .so.md5 and .so.sha1 files and you're done.


I would think that you'd need whatever protocol Maven uses to actually
put files on the repository. I guess the Maven protocol is simpler
than I thought and simply uses webdav authentication and the mvn
file::deploy simply calculates the URL for you.

That's good to know. That means you can replace all of Maven with some
fairly simple shell scripts using curl. Then in a non-Java project,
all the Maven repositories do is provide a nice interface for
searching and administration of a HTTP based repository.


Is there a maven-for-dummies reference somewhere that would make a good 
starting point for how the repository is supposed to work?  I tend to 
get lost easily with java-like things that have a bazillion separate 
config files all over the place.  Also, is mirroring/redundancy built 
into the design?


--
  Les Mikesell
   lesmikes...@gmail.com

Managing modifications to an open source product

[dan nessett]

I develop for a site that uses Mediawiki (MW). We make some modifications to it 
before deployment. Generally, (using subversion) we check out a tagged version 
into a workspace, recursively delete the .svn directories, modify a small 
number of files, add some of our own extensions, and then commit the result 
into our own repository. We then work with the source from there.

This approach means we have to track MW bug-fixes and add them to our modified 
version. I was wondering if there is a better way to accomplish the same 
objective. For example, we can use the svn:externals property to point to the 
MW repository version of the extensions we use, so each time they are updated, 
all we need to do is svn up on the externals directory.

The main source is a different story. Since we modify some of the files (and 
have no commit privileges to the MW repository), the files we modify are not 
within our purview to change (and understandably the MW people wouldn't allow 
it even if we had commit privileges).

Is there any way to use the svn:externals property to solve the main source 
issue? For example, could we point the revision we keep in our main repository 
to the correct revision in the MW repository and then tag the appropriate 
directories that contain the files we modify with svn:external. These latter 
svn:external properties would name the individual files we modify and point to 
the modified version that we could keep in our repository. My concern is we are 
"overloading" the files in the MW repository with files in our repository and I 
am not sure subversion allows that.

Regards,

Dan Nessett


  

RE: Managing modifications to an open source product

[Bob Archer]

> I develop for a site that uses Mediawiki (MW). We make some
> modifications to it before deployment. Generally, (using
> subversion) we check out a tagged version into a workspace,
> recursively delete the .svn directories, modify a small number of
> files, add some of our own extensions, and then commit the result
> into our own repository. We then work with the source from there.
> 
> This approach means we have to track MW bug-fixes and add them to
> our modified version. I was wondering if there is a better way to
> accomplish the same objective. For example, we can use the
> svn:externals property to point to the MW repository version of the
> extensions we use, so each time they are updated, all we need to do
> is svn up on the externals directory.
> 
> The main source is a different story. Since we modify some of the
> files (and have no commit privileges to the MW repository), the
> files we modify are not within our purview to change (and
> understandably the MW people wouldn't allow it even if we had
> commit privileges).
> 
> Is there any way to use the svn:externals property to solve the
> main source issue? For example, could we point the revision we keep
> in our main repository to the correct revision in the MW repository
> and then tag the appropriate directories that contain the files we
> modify with svn:external. These latter svn:external properties
> would name the individual files we modify and point to the modified
> version that we could keep in our repository. My concern is we are
> "overloading" the files in the MW repository with files in our
> repository and I am not sure subversion allows that.
> 

There is a whole section in the svn book about this... 

http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.advanced.vendorbr

BOb

Re: Managing modifications to an open source product

[Dan Nessett]

On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote:

>> I develop for a site that uses Mediawiki (MW). We make some
>> modifications to it before deployment. Generally, (using subversion) we
>> check out a tagged version into a workspace, recursively delete the
>> .svn directories, modify a small number of files, add some of our own
>> extensions, and then commit the result into our own repository. We then
>> work with the source from there.
>> 
>> This approach means we have to track MW bug-fixes and add them to our
>> modified version. I was wondering if there is a better way to
>> accomplish the same objective. For example, we can use the
>> svn:externals property to point to the MW repository version of the
>> extensions we use, so each time they are updated, all we need to do is
>> svn up on the externals directory.
>> 
>> The main source is a different story. Since we modify some of the files
>> (and have no commit privileges to the MW repository), the files we
>> modify are not within our purview to change (and understandably the MW
>> people wouldn't allow it even if we had commit privileges).
>> 
>> Is there any way to use the svn:externals property to solve the main
>> source issue? For example, could we point the revision we keep in our
>> main repository to the correct revision in the MW repository and then
>> tag the appropriate directories that contain the files we modify with
>> svn:external. These latter svn:external properties would name the
>> individual files we modify and point to the modified version that we
>> could keep in our repository. My concern is we are "overloading" the
>> files in the MW repository with files in our repository and I am not
>> sure subversion allows that.
>> 
>> 
> There is a whole section in the svn book about this...
> 
> http://svnbook.red-bean.com/nightly/en/svn-
book.html#svn.advanced.vendorbr
> 
> BOb

I have read this section. It is about vendor drops, but it doesn't answer 
the question I asked. Basically, we are doing a vendor drop now.

Dan



-- 
-- Dan Nessett

Re: Managing modifications to an open source product

[Olivier Sannier]

 On 14/10/2010 21:45, Dan Nessett wrote:

On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote:


I develop for a site that uses Mediawiki (MW). We make some
modifications to it before deployment. Generally, (using subversion) we
check out a tagged version into a workspace, recursively delete the
.svn directories, modify a small number of files, add some of our own
extensions, and then commit the result into our own repository. We then
work with the source from there.

This approach means we have to track MW bug-fixes and add them to our
modified version. I was wondering if there is a better way to
accomplish the same objective. For example, we can use the
svn:externals property to point to the MW repository version of the
extensions we use, so each time they are updated, all we need to do is
svn up on the externals directory.

The main source is a different story. Since we modify some of the files
(and have no commit privileges to the MW repository), the files we
modify are not within our purview to change (and understandably the MW
people wouldn't allow it even if we had commit privileges).

Is there any way to use the svn:externals property to solve the main
source issue? For example, could we point the revision we keep in our
main repository to the correct revision in the MW repository and then
tag the appropriate directories that contain the files we modify with
svn:external. These latter svn:external properties would name the
individual files we modify and point to the modified version that we
could keep in our repository. My concern is we are "overloading" the
files in the MW repository with files in our repository and I am not
sure subversion allows that.



There is a whole section in the svn book about this...

http://svnbook.red-bean.com/nightly/en/svn-

book.html#svn.advanced.vendorbr

BOb

I have read this section. It is about vendor drops, but it doesn't answer
the question I asked. Basically, we are doing a vendor drop now.

Dan



It actually does, but it's a bit hard to understand. Reread it carefully 
and you'll see that you actually track the public changes in a branch 
that you later merge back into your trunk.

Re: Repository organization for complex project

[David Weintraub]

What? You want a GOOD Maven manual? Real programmers don't use
manuals. We hack away for years in frustration and futility until we
die.

Maven is one of the WORST Unix documented projects I've seen. I know
of only two books: Maven: The Definitive Guide
 and Better Builds with
Maven . Better Builds with Maven
was full of errors both grammatical and technical. Maven: The
Definitive Guide is a bit better.

(Do not use "Maven, A Developer's Notebook" because this is for an
obsolete version of Maven. It's like learning Unix administration from
a MS-DOS 3.1 manual.)

The documentation on the Maven website itself is awful. It looks like
someone said "We have a Wiki
! Documentation
complete!

Fortunately, you don't really have to know too much about Maven if
you're not a Java developer. The intricacies of the pom.xml file don't
concern you. Possibly, the only Maven command you really have to know
is "mvn deploy:deploy-file" which doesn't require a pom.xml file
.
And, if that command concerns you too much, you can simply revert to
"curl".

You do have to understand the Maven repository layout which is a
fairly straight forward hierarchal affair and both Nexus and
Artifactory cover that pretty well.


On Thu, Oct 14, 2010 at 2:42 PM, Les Mikesell  wrote:
> On 10/14/2010 8:24 AM, David Weintraub wrote:
>>
>> On Thu, Oct 14, 2010 at 3:39 AM, Stephen Connolly
>>   wrote:
>>>
>>> Our C/C++ guys just use curl to POST the binaries to Nexus over
>>> http... we also POST the .pom file and the .md5 and .sha1 files...
>>> that is because one of their build toolchain envs cannot have Java on
>>> it... Nexus will rebuild the metadata.xml files for you, so all you
>>> really need is to post the .pom and the e.g. .so and the .pom.md5,
>>> .pom.sha1, .so.md5 and .so.sha1 files and you're done.
>>
>> I would think that you'd need whatever protocol Maven uses to actually
>> put files on the repository. I guess the Maven protocol is simpler
>> than I thought and simply uses webdav authentication and the mvn
>> file::deploy simply calculates the URL for you.
>>
>> That's good to know. That means you can replace all of Maven with some
>> fairly simple shell scripts using curl. Then in a non-Java project,
>> all the Maven repositories do is provide a nice interface for
>> searching and administration of a HTTP based repository.
>
> Is there a maven-for-dummies reference somewhere that would make a good
> starting point for how the repository is supposed to work?  I tend to get
> lost easily with java-like things that have a bazillion separate config
> files all over the place.  Also, is mirroring/redundancy built into the
> design?
>
> --
>  Les Mikesell
>   lesmikes...@gmail.com
>



-- 
David Weintraub
qazw...@gmail.com

Re: Managing modifications to an open source product

[Stefan Sperling]

On Thu, Oct 14, 2010 at 09:55:08PM +0200, Olivier Sannier wrote:
>  On 14/10/2010 21:45, Dan Nessett wrote:
> >On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote:
> >>>Generally, (using subversion) we
> >>>check out a tagged version into a workspace, recursively delete the
> >>>.svn directories, modify a small number of files, add some of our own
> >>>extensions, and then commit the result into our own repository. We then
> >>>work with the source from there.

> >>There is a whole section in the svn book about this...
> >>
> >>http://svnbook.red-bean.com/nightly/en/svn-book.html#svn.advanced.vendorbr

> >I have read this section. It is about vendor drops, but it doesn't answer
> >the question I asked. Basically, we are doing a vendor drop now.

It doesn't sound like you're using the approach described in the book.

With vendor branching, you work with 2 distinct branches, one containing
pristine upstream vendor code, and one containing the modified version.
Rather than adding the upstream source code mixed up with your own
modifications directly.

You might also want to take a look at Piston: http://piston.rubyforge.org/

Stefan

Re: Managing modifications to an open source product

[Dan Nessett]

On Thu, 14 Oct 2010 22:33:04 +0200, Stefan Sperling wrote:

> On Thu, Oct 14, 2010 at 09:55:08PM +0200, Olivier Sannier wrote:
>>  On 14/10/2010 21:45, Dan Nessett wrote:
>> >On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote:
>> >>>Generally, (using subversion) we
>> >>>check out a tagged version into a workspace, recursively delete the
>> >>>.svn directories, modify a small number of files, add some of our
>> >>>own extensions, and then commit the result into our own repository.
>> >>>We then work with the source from there.
> 
>> >>There is a whole section in the svn book about this...
>> >>
>> >>http://svnbook.red-bean.com/nightly/en/svn-
book.html#svn.advanced.vendorbr
> 
>> >I have read this section. It is about vendor drops, but it doesn't
>> >answer the question I asked. Basically, we are doing a vendor drop
>> >now.
> 
> It doesn't sound like you're using the approach described in the book.
> 
> With vendor branching, you work with 2 distinct branches, one containing
> pristine upstream vendor code, and one containing the modified version.
> Rather than adding the upstream source code mixed up with your own
> modifications directly.
> 
> You might also want to take a look at Piston:
> http://piston.rubyforge.org/
> 
> Stefan

Yes, you're right. We don't store an exact copy of each drop in our 
repository. Instead, we copy the drop into a workspace (effectively 
importing by checking out and then recursively deleting the .svn 
directories), merge there and commit to the new version. We couldn't 
understand why we need to keep an exact copy of the vendor drop in our 
repository.

Piston looks interesting, but it also appears to require ruby. This isn't 
a show stopper, but it would be nice if we could do what we want to do 
with subversion only. If that isn't possible, or it is too complex, then 
Piston might be the right way to go.

Dan



-- 
-- Dan Nessett

Re: svn Farm

[David Brodbeck]

On Thu, Oct 14, 2010 at 11:28 AM, Stefan Sperling  wrote:
> As of 1.6, Subversion asks the user before saving passwords in
> plaintext. 1.6 also added support for using GNOME Keyring and KDE Wallet
> as password stores.

Yup.  There are, as noted, unfortunately a lot of hassles involved
with those tools in a non-GUI environment; what we really need is a
lightweight, secure, standard keyring service.  But getting Linux
distros to standardize on *anything* is like herding cats, so I'm not
holding my breath. ;)  The assumption seems to be that these are
things that only desktop users really want, so bundling them as part
of the GUI is sufficient.  I don't blame Subversion for that, though.

-- 
David Brodbeck
System Administrator, Linguistics
University of Washington

Re: Repository organization for complex project

[Les Mikesell]

On 10/14/2010 3:11 PM, David Weintraub wrote:

What? You want a GOOD Maven manual? Real programmers don't use
manuals.


Yeah, I know - they don't write them either (except for subversion, of 
course).  As you might guess, I'm more of a system administrator than a 
programmer...



Fortunately, you don't really have to know too much about Maven if
you're not a Java developer. The intricacies of the pom.xml file don't
concern you. Possibly, the only Maven command you really have to know
is "mvn deploy:deploy-file" which doesn't require a pom.xml file
.
And, if that command concerns you too much, you can simply revert to
"curl".

You do have to understand the Maven repository layout which is a
fairly straight forward hierarchal affair and both Nexus and
Artifactory cover that pretty well.


Thanks - the piece I need is how to map our project tags and 
branch/revision_numbers into the maven repo URLs, hopefully in a way 
that wouldn't break it for actual maven use since we do have some java 
developers here.


--
  Les Mikesell
   lesmikes...@gmail.com

Re: Managing modifications to an open source product

[Dan Nessett]

On Thu, 14 Oct 2010 21:55:08 +0200, Olivier Sannier wrote:

> On 14/10/2010 21:45, Dan Nessett wrote:
>> On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote:
>>
 I develop for a site that uses Mediawiki (MW). We make some
 modifications to it before deployment. Generally, (using subversion)
 we check out a tagged version into a workspace, recursively delete
 the .svn directories, modify a small number of files, add some of our
 own extensions, and then commit the result into our own repository.
 We then work with the source from there.

 This approach means we have to track MW bug-fixes and add them to our
 modified version. I was wondering if there is a better way to
 accomplish the same objective. For example, we can use the
 svn:externals property to point to the MW repository version of the
 extensions we use, so each time they are updated, all we need to do
 is svn up on the externals directory.

 The main source is a different story. Since we modify some of the
 files (and have no commit privileges to the MW repository), the files
 we modify are not within our purview to change (and understandably
 the MW people wouldn't allow it even if we had commit privileges).

 Is there any way to use the svn:externals property to solve the main
 source issue? For example, could we point the revision we keep in our
 main repository to the correct revision in the MW repository and then
 tag the appropriate directories that contain the files we modify with
 svn:external. These latter svn:external properties would name the
 individual files we modify and point to the modified version that we
 could keep in our repository. My concern is we are "overloading" the
 files in the MW repository with files in our repository and I am not
 sure subversion allows that.


>>> There is a whole section in the svn book about this...
>>>
>>> http://svnbook.red-bean.com/nightly/en/svn-
>> book.html#svn.advanced.vendorbr
>>> BOb
>> I have read this section. It is about vendor drops, but it doesn't
>> answer the question I asked. Basically, we are doing a vendor drop now.
>>
>> Dan
> 
> 
> It actually does, but it's a bit hard to understand. Reread it carefully
> and you'll see that you actually track the public changes in a branch
> that you later merge back into your trunk.

I just reread the Vendor drop section in the book. It doesn't actually 
address the question I raised (which is about overloading files obtained 
with the svn:external definition). We don't do exactly what is described 
in the book, but logically it is the same. Instead of importing the 
vendor drop into the repository, we import it into a workspace (manually 
by checking it out and then recursively deleting the .svn directories. 
Perhaps we should use the svn import command, but that is a side issue). 
We then do the merge in the workspace and commit to a new version in the 
repository.

My question has to do with bypassing the fetch of the complete new MW 
version into our repository. (NB: MW revisions run about 150MB). Let me 
describe this in more detail.

+ When we wish to upgrade to a new MW revision, check out the directory 
in *our* repository that contains our modified files (and only those 
files).

+ Merge each of these files with those in the new MW revision. (NB: there 
are only a very few of these).

+ After resolving any conflicts, commit these changes into a new revision 
in *our* repository, probably tagging it so it is easy to reference.

+ svn mkdir to create an empty workspace.

+ With svn propedit add an svn:externals property to this directory so 
that the whole source effectively is an external fetch of the appropriate 
MW revision.

+ With svn propedit add lines to the svn:externals definition in the top-
level directory that describe the files we just merged, instructing 
subversion to get these from the new branch we just created in *our* 
repository. (Note: this is where the "overload" occurs, since files with 
the same name and at the same location in the MW repository revision 
exist. We don't want fetch these - we want a checkout to get the merged 
files from *our* repository instead.)

+ Commit this to *our* repository under a new tag.

Will this work? If so, are there some reasons why it isn't a good idea.

-- 
-- Dan Nessett

Re: svn Farm

[Stefan Sperling]

On Thu, Oct 14, 2010 at 01:58:19PM -0700, David Brodbeck wrote:
> On Thu, Oct 14, 2010 at 11:28 AM, Stefan Sperling  wrote:
> > As of 1.6, Subversion asks the user before saving passwords in
> > plaintext. 1.6 also added support for using GNOME Keyring and KDE Wallet
> > as password stores.
> 
> Yup.  There are, as noted, unfortunately a lot of hassles involved
> with those tools in a non-GUI environment; what we really need is a
> lightweight, secure, standard keyring service.  But getting Linux
> distros to standardize on *anything* is like herding cats, so I'm not
> holding my breath. ;)  The assumption seems to be that these are
> things that only desktop users really want, so bundling them as part
> of the GUI is sufficient.  I don't blame Subversion for that, though.

I hope the work-in-progress gpg-agent support I mentioned will fill that gap.
Would using gpg-agent work for you?

Stefan

Re: Managing modifications to an open source product

[Dan Nessett]

On Thu, 14 Oct 2010 21:17:59 +, Dan Nessett wrote:

> On Thu, 14 Oct 2010 21:55:08 +0200, Olivier Sannier wrote:
> 
>> On 14/10/2010 21:45, Dan Nessett wrote:
>>> On Thu, 14 Oct 2010 15:38:41 -0400, Bob Archer wrote:
>>>
> I develop for a site that uses Mediawiki (MW). We make some
> modifications to it before deployment. Generally, (using subversion)
> we check out a tagged version into a workspace, recursively delete
> the .svn directories, modify a small number of files, add some of
> our own extensions, and then commit the result into our own
> repository. We then work with the source from there.
>
> This approach means we have to track MW bug-fixes and add them to
> our modified version. I was wondering if there is a better way to
> accomplish the same objective. For example, we can use the
> svn:externals property to point to the MW repository version of the
> extensions we use, so each time they are updated, all we need to do
> is svn up on the externals directory.
>
> The main source is a different story. Since we modify some of the
> files (and have no commit privileges to the MW repository), the
> files we modify are not within our purview to change (and
> understandably the MW people wouldn't allow it even if we had commit
> privileges).
>
> Is there any way to use the svn:externals property to solve the main
> source issue? For example, could we point the revision we keep in
> our main repository to the correct revision in the MW repository and
> then tag the appropriate directories that contain the files we
> modify with svn:external. These latter svn:external properties would
> name the individual files we modify and point to the modified
> version that we could keep in our repository. My concern is we are
> "overloading" the files in the MW repository with files in our
> repository and I am not sure subversion allows that.
>
>
 There is a whole section in the svn book about this...

 http://svnbook.red-bean.com/nightly/en/svn-
>>> book.html#svn.advanced.vendorbr
 BOb
>>> I have read this section. It is about vendor drops, but it doesn't
>>> answer the question I asked. Basically, we are doing a vendor drop
>>> now.
>>>
>>> Dan
>> 
>> 
>> It actually does, but it's a bit hard to understand. Reread it
>> carefully and you'll see that you actually track the public changes in
>> a branch that you later merge back into your trunk.
> 
> I just reread the Vendor drop section in the book. It doesn't actually
> address the question I raised (which is about overloading files obtained
> with the svn:external definition). We don't do exactly what is described
> in the book, but logically it is the same. Instead of importing the
> vendor drop into the repository, we import it into a workspace (manually
> by checking it out and then recursively deleting the .svn directories.
> Perhaps we should use the svn import command, but that is a side issue).
> We then do the merge in the workspace and commit to a new version in the
> repository.
> 
> My question has to do with bypassing the fetch of the complete new MW
> version into our repository. (NB: MW revisions run about 150MB). Let me
> describe this in more detail.
> 
> + When we wish to upgrade to a new MW revision, check out the directory
> in *our* repository that contains our modified files (and only those
> files).
> 
> + Merge each of these files with those in the new MW revision. (NB:
> there are only a very few of these).
> 
> + After resolving any conflicts, commit these changes into a new
> revision in *our* repository, probably tagging it so it is easy to
> reference.
> 
> + svn mkdir to create an empty workspace.
> 
> + With svn propedit add an svn:externals property to this directory so
> that the whole source effectively is an external fetch of the
> appropriate MW revision.
> 
> + With svn propedit add lines to the svn:externals definition in the
> top- level directory that describe the files we just merged, instructing
> subversion to get these from the new branch we just created in *our*
> repository. (Note: this is where the "overload" occurs, since files with
> the same name and at the same location in the MW repository revision
> exist. We don't want fetch these - we want a checkout to get the merged
> files from *our* repository instead.)
> 
> + Commit this to *our* repository under a new tag.
> 
> Will this work? If so, are there some reasons why it isn't a good idea.

Bad etiquette to answer my own question, but I just reread the Externals 
Definition section of the book and found this:

"An externals definition can point only to directories, not to files." 
So, what I have described won't work even if overloading with 
svn:externals is supported.

-- 
-- Dan Nessett

Re: Managing modifications to an open source product

[Geoff Hoffman]

Dan, 

This isn't that much help but are you aware that svn export is exactly the same 
as svn checkout & run a script to delete all the .svn folders ...? The problem 
with stripping .svn from vendor checkout is then you cannot update it; you're 
forced to export the whole tree every time.

The method of having a working copy somewhere pointing to the vendor files, 
that you can svn update whenever you like, and then diff the changes into your 
tree is the easiest way to accomplish what you want, I think. 

What I would recommend is that you have two branches for this (plus trunk of 
your code):

[your stuff] => trunk-working-copy
[their stuff] => from-vendor-svn
[intermediate] = svn copy [your stuff -r HEAD]

Then at any time you can: 

svn up [their stuff]
svn copy [your stuff] => [intermediate]
(so you can do your diff/merge to an intermediate branch)
test on [intermediate] 
if it passes
merge it to trunk, delete [intermediate]

Bear in mind, not sure what OS you're on, but e.g. on Windows using Araxis 
merge, merging two or three trees is far (several orders of magnitude) simpler 
than using svn diff.

HTH,

Geoff

Re: Managing modifications to an open source product

[Stefan Sperling]

On Thu, Oct 14, 2010 at 09:17:59PM +, Dan Nessett wrote:
> My question has to do with bypassing the fetch of the complete new MW 
> version into our repository. (NB: MW revisions run about 150MB). Let me 
> describe this in more detail.
> 
> + When we wish to upgrade to a new MW revision, check out the directory 
> in *our* repository that contains our modified files (and only those 
> files).
> 
> + Merge each of these files with those in the new MW revision. (NB: there 
> are only a very few of these).
> 
> + After resolving any conflicts, commit these changes into a new revision 
> in *our* repository, probably tagging it so it is easy to reference.
> 
> + svn mkdir to create an empty workspace.
> 
> + With svn propedit add an svn:externals property to this directory so 
> that the whole source effectively is an external fetch of the appropriate 
> MW revision.
> 
> + With svn propedit add lines to the svn:externals definition in the top-
> level directory that describe the files we just merged, instructing 
> subversion to get these from the new branch we just created in *our* 
> repository. (Note: this is where the "overload" occurs, since files with 
> the same name and at the same location in the MW repository revision 
> exist. We don't want fetch these - we want a checkout to get the merged 
> files from *our* repository instead.)
>
> + Commit this to *our* repository under a new tag.
> 
> Will this work?

Unfortunately, this won't work. You cannot use externals like this.
The existing files are in the way.

But I may be wrong, or maybe I misunderstand your idea.
In general, with a complex tool like Subversion the best way to see
if something will work is to just try it out and see.

> If so, are there some reasons why it isn't a good idea.

There's nothing wrong with your idea itself.
But Subversion doesn't support an "overlay" concept like you want
to realise with externals.

Also, I'd recommend not using file externals for anything, because
their implementation is a hack and there are several known issues:
http://subversion.tigris.org/issues/show_bug.cgi?id=3351
http://subversion.tigris.org/issues/show_bug.cgi?id=3589
http://subversion.tigris.org/issues/show_bug.cgi?id=3665
http://subversion.tigris.org/issues/show_bug.cgi?id=3563
Directory externals are fine though.

The problem you're trying to solve (vendor branching) isn't easy,
and many have tried before and devised solutions.
So I'd suggest that before trying to devise schemes of your own,
shop around for existing solutions. One of them will probably fit the bill.

The existing alternatives I know about are:

  - Subversion's vendor branching with svn_load_dirs.
This works fine in general, but handling of renamed files is a bit
clunky. It's what the svnbook describes.

  - Play with Subversion's foreign repository merges.
This is a little-known feature that could help you track the
MediaWiki changes, as long as MediaWiki keeps using Subversion.
See http://blogs.open.collab.net/svn/2008/03/do-not-post-mer.html
(which was written before the 1.5 release but is still relevant).
You'd do something like this in the working copy of your custom
MW code:
  svn merge \
http://svn.wikimedia.org/svnroot/mediawiki/trunk/@74770 \
http://svn.wikimedia.org/svnroot/mediawiki/trunk/@74798

  - Try Piston. I've never tried it myself, but its interface seems nice
and I've seen it being recommended on this list before. The dependency
on Ruby isn't nice but quite possibly worth it.

Good luck,
Stefan

Re: Managing modifications to an open source product

[Stefan Sperling]

On Thu, Oct 14, 2010 at 09:48:21PM +, Dan Nessett wrote:
> Bad etiquette to answer my own question, but I just reread the Externals 
> Definition section of the book and found this:
> 
> "An externals definition can point only to directories, not to files." 
> So, what I have described won't work even if overloading with 
> svn:externals is supported.

You're probably reading an outdated version of the book.
See the nightly edition: 
http://svnbook.red-bean.com/nightly/en/svn.advanced.externals.html
File externals are indeed supported as of 1.6, with the caveats I
listed in my other reply (those problems aren't documented in the book).
The current plan is to fix file externals in the upcoming 1.7 release.

But, as I mentioned, externals cannot be used as "overlays".

Stefan

Re: svn Farm

[David Brodbeck]

On Thu, Oct 14, 2010 at 2:22 PM, Stefan Sperling  wrote:
> I hope the work-in-progress gpg-agent support I mentioned will fill that gap.
> Would using gpg-agent work for you?

Quite likely.  I haven't really played with gpg-agent, but I don't
know of any reason it shouldn't work.


-- 
David Brodbeck
System Administrator, Linguistics
University of Washington

Re: Repository organization for complex project

[Stephen Connolly]

On 14 October 2010 22:03, Les Mikesell  wrote:
> On 10/14/2010 3:11 PM, David Weintraub wrote:
>>
>> What? You want a GOOD Maven manual? Real programmers don't use
>> manuals.
>
> Yeah, I know - they don't write them either (except for subversion, of
> course).  As you might guess, I'm more of a system administrator than a
> programmer...
>
>> Fortunately, you don't really have to know too much about Maven if
>> you're not a Java developer. The intricacies of the pom.xml file don't
>> concern you. Possibly, the only Maven command you really have to know
>> is "mvn deploy:deploy-file" which doesn't require a pom.xml file
>>
>> .
>> And, if that command concerns you too much, you can simply revert to
>> "curl".
>>
>> You do have to understand the Maven repository layout which is a
>> fairly straight forward hierarchal affair and both Nexus and
>> Artifactory cover that pretty well.
>
> Thanks - the piece I need is how to map our project tags and
> branch/revision_numbers into the maven repo URLs, hopefully in a way that
> wouldn't break it for actual maven use since we do have some java developers
> here.

every artifact in the maven repository has a set of coordinates, known
as GAV (though technically it is GAVCT)

GroupId:ArtifactId:Version[:Classifier]:Type

in the group id, you use reverse DNS name to establish ownership of
the namespace, dot's are replaced by slashes... so one project I am
working on at github uses the groupId
com.github.stephenc.java-iso-tools as the project url is
http://github.com/stephenc/java-iso-tools

I have other things where I use groupIds starting with com.one-dash
because I own the domain.

The artifactId can be whatever you like (within reason)

The version is a version number

The classifier is similar to artifactId and this is more for
side-artifacts... you probably don't want classifiers in most cases...
side-artifacts would be things like javadocs... try to keep one GAV
for one theing that you build, e.g. a separate GAV for the x86 and the
ARM versions of your build rather than abuse classifiers [so that you
play better with the java builds]

to consturuct the url

URL=${REPO_ROOT}/$(echo ${GROUP_ID} | sed -e
"s:\.:/:g")/$ARTIFACT_ID/$VERSION/$ARTIFACT_ID-${CLASSIFIER}-${VERSION}.${TYPE}

if you have a classifier and

URL=${REPO_ROOT}/$(echo ${GROUP_ID} | sed -e
"s:\.:/:g")/$ARTIFACT_ID/$VERSION/$ARTIFACT_ID-${VERSION}.${TYPE}

if you don't

-Stephen

>
> --
>  Les Mikesell
>   lesmikes...@gmail.com
>

Re: Managing modifications to an open source product

[Dan Nessett]

On Fri, 15 Oct 2010 00:08:28 +0200, Stefan Sperling wrote:

> On Thu, Oct 14, 2010 at 09:17:59PM +, Dan Nessett wrote:
>> My question has to do with bypassing the fetch of the complete new MW
>> version into our repository. (NB: MW revisions run about 150MB). Let me
>> describe this in more detail.
>> 
>> + When we wish to upgrade to a new MW revision, check out the directory
>> in *our* repository that contains our modified files (and only those
>> files).
>> 
>> + Merge each of these files with those in the new MW revision. (NB:
>> there are only a very few of these).
>> 
>> + After resolving any conflicts, commit these changes into a new
>> revision in *our* repository, probably tagging it so it is easy to
>> reference.
>> 
>> + svn mkdir to create an empty workspace.
>> 
>> + With svn propedit add an svn:externals property to this directory so
>> that the whole source effectively is an external fetch of the
>> appropriate MW revision.
>> 
>> + With svn propedit add lines to the svn:externals definition in the
>> top- level directory that describe the files we just merged,
>> instructing subversion to get these from the new branch we just created
>> in *our* repository. (Note: this is where the "overload" occurs, since
>> files with the same name and at the same location in the MW repository
>> revision exist. We don't want fetch these - we want a checkout to get
>> the merged files from *our* repository instead.)
>>
>> + Commit this to *our* repository under a new tag.
>> 
>> Will this work?
> 
> Unfortunately, this won't work. You cannot use externals like this. The
> existing files are in the way.
> 
> But I may be wrong, or maybe I misunderstand your idea. In general, with
> a complex tool like Subversion the best way to see if something will
> work is to just try it out and see.
> 
>> If so, are there some reasons why it isn't a good idea.
> 
> There's nothing wrong with your idea itself. But Subversion doesn't
> support an "overlay" concept like you want to realise with externals.
> 
> Also, I'd recommend not using file externals for anything, because their
> implementation is a hack and there are several known issues:
> http://subversion.tigris.org/issues/show_bug.cgi?id=3351
> http://subversion.tigris.org/issues/show_bug.cgi?id=3589
> http://subversion.tigris.org/issues/show_bug.cgi?id=3665
> http://subversion.tigris.org/issues/show_bug.cgi?id=3563 Directory
> externals are fine though.
> 
> The problem you're trying to solve (vendor branching) isn't easy, and
> many have tried before and devised solutions. So I'd suggest that before
> trying to devise schemes of your own, shop around for existing
> solutions. One of them will probably fit the bill.
> 
> The existing alternatives I know about are:
> 
>   - Subversion's vendor branching with svn_load_dirs.
> This works fine in general, but handling of renamed files is a bit
> clunky. It's what the svnbook describes.
> 
>   - Play with Subversion's foreign repository merges.
> This is a little-known feature that could help you track the
> MediaWiki changes, as long as MediaWiki keeps using Subversion. See
> http://blogs.open.collab.net/svn/2008/03/do-not-post-mer.html (which
> was written before the 1.5 release but is still relevant). You'd do
> something like this in the working copy of your custom MW code:
>   svn merge \
> http://svn.wikimedia.org/svnroot/mediawiki/trunk/@74770 \
> http://svn.wikimedia.org/svnroot/mediawiki/trunk/@74798
> 
>   - Try Piston. I've never tried it myself, but its interface seems nice
> and I've seen it being recommended on this list before. The
> dependency on Ruby isn't nice but quite possibly worth it.
> 
> Good luck,
> Stefan

Thanks. Your description of alternative approaches was very helpful.

Regards,

-- 
-- Dan Nessett