Error processing externals E205011 and W155035 with simple update
Hello, I am running SVN 1.8.1 under Win7. We have an extensive collection of file externals that we use to manage shared code between projects (hundreds of files per working directory). As known, the performance when performing an SVN Update is slow. To work around this, we're attempting to only update externals that are out of date (by comparing the local revisions of each file to what is in the svn:externals. As we iterate across the files, if we see an external out of date, we execute an svn update . Doing this, I'm starting to get corrupted externals I can reproduce this using the following the steps listed below. Even if I am performing operations out of order with what is expected, I'd expect that I'd be unable to harm or corrupt the working directory I basically create a clean checkout of a repository. Verify everything is cool, but realize one file external needs a new revision. I edit the property (using TSVN) and commit the property. I then update the working directory ignoring externals and only for the immediate directory (because external updates are slow). I then update the file I know is out of date (because I just edited the svn;externals). Ouch, I get an error and it reports that the file has been switched. The only way I can repair this is to delete the checkout and recreate it. I believe this also occurs under 1.7, but will have to confirm later. Can any confirm first that this is indeed an issue, and second how to resolve this? Thanks! Dan c:\Project_files\Test>svn status -v --depth immediates 2850 2850 dlellis . X 2818 2818 dlellis 1.txt X 2633 2633 dlellis 6.txt X 2626 2626 dlellis 7.txt 2850 2847 dlellis CLP 2850 2783 dlellis SM X 2812 2812 dlellis checkintest.txt X 2814 2814 dlellis commit.vbs X 2728 2728 dlellis project.conf c:\Project_files\Test>svn pg svn:externals http://test/pd_sandbox/plcm/swdb/M4000/project.conf@2728 project.conf http://test/pd_sandbox/plcm/swdb/Temp/6.txt@2633 6.txt http://test/pd_sandbox/plcm/swdb/Temp/7.txt@2626 7.txt http://test/pd_sandbox/plcm/swdb/Temp/1.txt@2818 1.txt http://test/pd_sandbox/plcm/swdb/Temp/commit.vbs@2814 commit.vbs http://test/pd_sandbox/plcm/swdb/Temp/checkintest.txt@2812 checkintest.txt * (Modify external here via TSVN) 1.txt from revision 2818 to 2827. c:\Project_files\Test>svn pg svn:externals http://test/pd_sandbox/plcm/swdb/M4000/project.conf@2728 project.conf http://test/pd_sandbox/plcm/swdb/Temp/6.txt@2633 6.txt http://test/pd_sandbox/plcm/swdb/Temp/7.txt@2626 7.txt http://test/pd_sandbox/plcm/swdb/Temp/1.txt@2827 1.txt http://test/pd_sandbox/plcm/swdb/Temp/commit.vbs@2814 commit.vbs http://test/pd_sandbox/plcm/swdb/Temp/checkintest.txt@2812 checkintest.txt c:\Project_files\Test>svn commit . --depth empty -m "checkin test" Sending. Committed revision 2851. c:\Project_files\Test>svn update --depth empty --ignore-externals Updating '.': At revision 2851. c:\Project_files\Test>svn update 1.txt Updating '1.txt': U1.txt Updated to revision 2851. c:\Project_files\Test>svn update Fetching external item into '1.txt': ***svn: warning: W155035: The specified path has an unexpected status At revision 2851. *** svn: E205011: Failure occurred processing one or more externals definitions c:\Project_files\Test>svn status -v --depth immediates 2851 2851 dlellis . ***S 2851 2827 dlellis 1.txt X 2633 2633 dlellis 6.txt X 2626 2626 dlellis 7.txt 2851 2847 dlellis CLP 2851 2783 dlellis SM X 2812 2812 dlellis checkintest.txt X 2814 2814 dlellis commit.vbs X 2728 2728 dlellis project.conf
How to change paths on an external file without a full update --depth infinity?
Hello everyone, In an attempt to work around the slow performance issues with externals, I'm trying to perform selective updates on external files without performing an "svn update --depth infinity". If I update the path on an external for foo.c to be from /bar1/ to /bar2/, and commit the property change, I'd hope that an "svn update foo.c" would result in foo.c being changed from /bar1/ to /bar2/, but it gets deleted instead. I understand why foo.c gets deleted (/bar1/foo.c is different pedigree than /bar2/foo.c despite the same file name), but would really hope to perform an "svn update foo.c" to get the latest version referenced by the svn:externals. How can I get svn to fetch me the file referenced in svn:externals without performing an "svn update --depth infinity"? Remind me again why externals are only processed when depth is infinity? Thanks! Dan
Re: Error processing externals E205011 and W155035 with simple update
Hello everyone, I have confirmed that this behavior also exists in SVN 1.7.5 on Win7. I appreciate any thoughts any one might have. Thanks, Dan From: dlel...@rockwellcollins.com To: users@subversion.apache.org Date: 08/09/2013 03:55 PM Subject:Error processing externals E205011 and W155035 with simple update Hello, I am running SVN 1.8.1 under Win7. We have an extensive collection of file externals that we use to manage shared code between projects (hundreds of files per working directory). As known, the performance when performing an SVN Update is slow. To work around this, we're attempting to only update externals that are out of date (by comparing the local revisions of each file to what is in the svn:externals. As we iterate across the files, if we see an external out of date, we execute an svn update . Doing this, I'm starting to get corrupted externals I can reproduce this using the following the steps listed below. Even if I am performing operations out of order with what is expected, I'd expect that I'd be unable to harm or corrupt the working directory I basically create a clean checkout of a repository. Verify everything is cool, but realize one file external needs a new revision. I edit the property (using TSVN) and commit the property. I then update the working directory ignoring externals and only for the immediate directory (because external updates are slow). I then update the file I know is out of date (because I just edited the svn;externals). Ouch, I get an error and it reports that the file has been switched. The only way I can repair this is to delete the checkout and recreate it. I believe this also occurs under 1.7, but will have to confirm later. Can any confirm first that this is indeed an issue, and second how to resolve this? Thanks! Dan c:\Project_files\Test>svn status -v --depth immediates 2850 2850 dlellis . X 2818 2818 dlellis 1.txt X 2633 2633 dlellis 6.txt X 2626 2626 dlellis 7.txt 2850 2847 dlellis CLP 2850 2783 dlellis SM X 2812 2812 dlellis checkintest.txt X 2814 2814 dlellis commit.vbs X 2728 2728 dlellis project.conf c:\Project_files\Test>svn pg svn:externals http://test/pd_sandbox/plcm/swdb/M4000/project.conf@2728 project.conf http://test/pd_sandbox/plcm/swdb/Temp/6.txt@2633 6.txt http://test/pd_sandbox/plcm/swdb/Temp/7.txt@2626 7.txt http://test/pd_sandbox/plcm/swdb/Temp/1.txt@2818 1.txt http://test/pd_sandbox/plcm/swdb/Temp/commit.vbs@2814 commit.vbs http://test/pd_sandbox/plcm/swdb/Temp/checkintest.txt@2812 checkintest.txt * (Modify external here via TSVN) 1.txt from revision 2818 to 2827. c:\Project_files\Test>svn pg svn:externals http://test/pd_sandbox/plcm/swdb/M4000/project.conf@2728 project.conf http://test/pd_sandbox/plcm/swdb/Temp/6.txt@2633 6.txt http://test/pd_sandbox/plcm/swdb/Temp/7.txt@2626 7.txt http://test/pd_sandbox/plcm/swdb/Temp/1.txt@2827 1.txt http://test/pd_sandbox/plcm/swdb/Temp/commit.vbs@2814 commit.vbs http://test/pd_sandbox/plcm/swdb/Temp/checkintest.txt@2812 checkintest.txt c:\Project_files\Test>svn commit . --depth empty -m "checkin test" Sending. Committed revision 2851. c:\Project_files\Test>svn update --depth empty --ignore-externals Updating '.': At revision 2851. c:\Project_files\Test>svn update 1.txt Updating '1.txt': U1.txt Updated to revision 2851. c:\Project_files\Test>svn update Fetching external item into '1.txt': ***svn: warning: W155035: The specified path has an unexpected status At revision 2851. *** svn: E205011: Failure occurred processing one or more externals definitions c:\Project_files\Test>svn status -v --depth immediates 2851 2851 dlellis . ***S 2851 2827 dlellis 1.txt X 2633 2633 dlellis 6.txt X 2626 2626 dlellis 7.txt 2851 2847 dlellis CLP 2851 2783 dlellis SM X 2812 2812 dlellis checkintest.txt X 2814 2814 dlellis commit.vbs X 2728 2728 dlellis project.conf
Re: How to change paths on an external file without a full update --depth infinity?
Hi Johan, I think you are spot on on both issues - thank you. The performance issue relates to SVN externals stem, in my case, from new connections for each http request for externals versus keeping an open one for non-externals. Each connection results in a second or so delay. Multiply this by hundreds and updates that would take seconds, now takes ten minutes. There's been quite a bit of discussion about pooling connections I believe, but I don't know the current state of this effort or if its in any plan. We're attempting to work around this by updating the subversion properties with svn update . --depth empty --ignore-externals and iterating over the externals and comparing to our local revision. If we see a difference, we update the file. This is the behavior I'd like to see externals take in the end (I'm sure there are good reasons for the current methodology). This issue and another issue that can corrupt the working copy are creating some barriers for us. Externals are becoming more and more important as we look to finer grained product line sharing of code. All that said, is there a work around to convincing SVN to recreate the file without a full fledge update? Thanks, Dan From: Johan Corveleyn To: dlel...@rockwellcollins.com Cc: "users@subversion.apache.org" Date: 08/10/2013 03:15 PM Subject:Re: How to change paths on an external file without a full update --depth infinity? On Sat, Aug 10, 2013 at 1:07 AM, wrote: > Hello everyone, > > In an attempt to work around the slow performance issues with externals, I'm > trying to perform selective updates on external files without performing an > "svn update --depth infinity". First: what "slow performance issues with externals"? Can you give some concrete numbers, or some references to other posts about this? What svn version are you using? Is this slowness a new problem? > If I update the path on an external for foo.c to be from /bar1/ to /bar2/, > and commit the property change, I'd hope that an "svn update foo.c" would > result in foo.c being changed from /bar1/ to /bar2/, but it gets deleted > instead. I understand why foo.c gets deleted (/bar1/foo.c is different > pedigree than /bar2/foo.c despite the same file name), but would really hope > to perform an "svn update foo.c" to get the latest version referenced by the > svn:externals. This looks similar to this issue: http://subversion.tigris.org/issues/show_bug.cgi?id=4001 (update fails to replace a file external with an actual node) It's about replacing a file external with a "normal" versioned node, in one update. But your description might be some other manifestation of the same problem ... not sure. > How can I get svn to fetch me the file referenced in svn:externals without > performing an "svn update --depth infinity"? > > Remind me again why externals are only processed when depth is infinity? I suppose this issue is the "reason": http://subversion.tigris.org/issues/show_bug.cgi?id=3311 (externals are not created unless depth=infinity) But I'm not sure (perhaps that issue is only about dir externals). -- Johan
Re: Error processing externals E205011 and W155035 with simple update
Hi Johan, I'll try and use the reproduction template, but will need to work this on Monday as it appears I'll need to work the template a bit to work with http://. Here's the quick recipe Create a file external in svn:externals and commit it and perform full update (foo.c rev 1). Create a new revision of the file and commit (foo.c rev 2) Edit the svn:externals to point to latest revision (foo.c rev 2) Now here's the magic to reproduce the issue: ;: Only perform an "svn update . -depth empty --ignore-externals" // this should bring in the new svn:externals property with foo.c rev 2 Only update the file as "svn update foo.c" This should now show the file as being switched with an "svn status -v foo.c": Performing a full "svn update" will cause the errors and seem to put the WC into a bad state. A full output of the command prompt is in the listing below. I appreciate the help! Dan From: Johan Corveleyn To: dlel...@rockwellcollins.com Cc: "users@subversion.apache.org" Date: 08/10/2013 03:40 PM Subject:Re: Error processing externals E205011 and W155035 with simple update Like I replied to your other post: please give us some more concrete information about this "known slowness of svn update"? That said, if you can corrupt your working copy by using a series of regular svn commands, that's clearly a bug. Would it be possible for you to write down a reproduction recipe starting from a clean repository? Or even better: a reproduction script, for example starting from this template: http://subversion.apache.org/docs/community-guide/repro-template.bat That would help a lot in understanding and reproducing the issue. -- Johan On Sat, Aug 10, 2013 at 3:58 AM, wrote: > Hello everyone, > > I have confirmed that this behavior also exists in SVN 1.7.5 on Win7. I > appreciate any thoughts any one might have. > > Thanks, > Dan > > > > > From:dlel...@rockwellcollins.com > To:users@subversion.apache.org > Date:08/09/2013 03:55 PM > Subject:Error processing externals E205011 and W155035 with simple > update > > > > > Hello, > > I am running SVN 1.8.1 under Win7. We have an extensive collection of file > externals that we use to manage shared code between projects (hundreds of > files per working directory). > > As known, the performance when performing an SVN Update is slow. To work > around this, we're attempting to only update externals that are out of date > (by comparing the local revisions of each file to what is in the > svn:externals. As we iterate across the files, if we see an external out of > date, we execute an svn update . > > Doing this, I'm starting to get corrupted externals I can reproduce > this using the following the steps listed below. Even if I am performing > operations out of order with what is expected, I'd expect that I'd be unable > to harm or corrupt the working directory > > > I basically create a clean checkout of a repository. > Verify everything is cool, but realize one file external needs a new > revision. > I edit the property (using TSVN) and commit the property. > I then update the working directory ignoring externals and only for the > immediate directory (because external updates are slow). > I then update the file I know is out of date (because I just edited the > svn;externals). > Ouch, I get an error and it reports that the file has been switched. > The only way I can repair this is to delete the checkout and recreate it. > > I believe this also occurs under 1.7, but will have to confirm later. > > Can any confirm first that this is indeed an issue, and second how to > resolve this? > > Thanks! > Dan > > > c:\Project_files\Test>svn status -v --depth immediates > > 2850 2850 dlellis . > > X 2818 2818 dlellis 1.txt > > X 2633 2633 dlellis 6.txt > > X 2626 2626 dlellis 7.txt > > 2850 2847 dlellis CLP > > 2850 2783 dlellis SM > > X 2812 2812 dlellis checkintest.txt > > X 2814 2814 dlellis commit.vbs > > X 2728 2728 dlellis project.conf > > c:\Project_files\Test>svn pg svn:externals > > http://test/pd_sandbox/plcm/swdb/M4000/project.conf@2728 project.conf > > http://test/pd_sandbox/plcm/swdb/Temp/6.txt@2633 6.txt > > http://test/pd_sandbox/plcm/swdb/Temp/7.txt@2626 7.txt > > http://test/pd_sandbox/plcm/swdb/Temp/1.txt@2818 1.txt > > http://test/pd_sandbox/plcm/swdb/Temp/
Re: Error processing externals E205011 and W155035 with simple update
Johan Corveleyn wrote on 08/11/2013 03:09:55 AM: > From: Johan Corveleyn > To: dlel...@rockwellcollins.com > Cc: "users@subversion.apache.org" > Date: 08/11/2013 03:10 AM > Subject: Re: Error processing externals E205011 and W155035 with simple update > > [ In general, this list prefers bottom-posting (i.e. putting your > reply at the bottom) or inline replying. More below ... ] Roger that! > > On Sun, Aug 11, 2013 at 12:56 AM, wrote: > > Hi Johan, > > > > I'll try and use the reproduction template, but will need to work this on > > Monday as it appears I'll need to work the template a bit to work with > > http://. > > I don't think the bug is related to the protocol. From what I've read > so far, I think you should be able to reproduce this with file:// just > as well (if not: that would be interesting additional information). > > Note that you can use "repository-relative externals" with ^/ etc, so > you don't have to specify the protocol in your externals definition. Sorry, I wasn't implying that the bug is HTTP related, just that our enterprise only allows http access to our SVN repos. That said, I'm also not sure that I can run the template as it appears to require creating a clean repo and we, as lowly developers, are not trusted with such power. > > > Here's the quick recipe > > > > Create a file external in svn:externals and commit it and perform full > > update (foo.c rev 1). > > Please give the exact value you put into the property. There are a > couple of variants of the syntax, and it's not clear whether you use a > peg revision etc. This is what SVN reports from an svn pg svn:externals before and after the update http://test/pd_sandbox/plcm/swdb/Temp/1.txt@2818 1.txt http://test/pd_sandbox/plcm/swdb/Temp/1.txt@2827 1.txt This issue appears if I either use TSVN to enter the above to enter the local path, URL, and peg revision OR if I programatically perform this through command line SVN as in : svn propset svn:externals -F "tmp81A7.tmp" "C:\Project_files\test" where the contents of the tmp file are: http://test/pd_sandbox/plcm/swdb/Test/project.conf@2728 project.conf http://test/pd_sandbox/plcm/swdb/Temp/6.txt@2633 6.txt http://test/pd_sandbox/plcm/swdb/Temp/7.txt@2626 7.txt http://test/pd_sandbox/plcm/swdb/Temp/1.txt@2827 1.txt http://test/pd_sandbox/plcm/swdb/Temp/commit.vbs@2814 commit.vbs http://test/pd_sandbox/plcm/swdb/Temp/checkintest.txt@2812 checkintest.txt What is really interesting (to me at least), is that SVN appears to think the file is then switched (shows S with an svn info 1.txt) c:\Project_files\Test>svn status -v --depth immediates 2851 2851 dlellis . S 2851 2827 dlellis 1.txt > > > Create a new revision of the file and commit (foo.c rev 2) > > Edit the svn:externals to point to latest revision (foo.c rev 2) > > And the exact syntax of the new value. > > > > > Now here's the magic to reproduce the issue: > > ;: > > Only perform an "svn update . -depth empty --ignore-externals" // this > > should bring in the new svn:externals property with foo.c rev 2 > > Only update the file as "svn update foo.c" > > > > This should now show the file as being switched with an "svn status -v > > foo.c": > > > > Performing a full "svn update" will cause the errors and seem to put the WC > > into a bad state. > > > -- > Johan Thanks, Dan
Re: How to change paths on an external file without a full update --depth infinity?
Johan Corveleyn wrote on 08/11/2013 03:24:31 AM: > > Hi Johan, > > > > I think you are spot on on both issues - thank you. > > You're welcome :-). > > > The performance issue relates to SVN externals stem, in my case, from new > > connections for each http request for externals versus keeping an open one > > for non-externals. Each connection results in a second or so delay. > > Multiply this by hundreds and updates that would take seconds, nowtakes ten > > minutes. There's been quite a bit of discussion about pooling connections I > > believe, but I don't know the current state of this effort or if its in any > > plan. > > Hm, there were indeed some discussions (and patch attempts) recently, > but I'm not sure of the current state either. There is an open issue > about this in the issue tracker: > > http://subversion.tigris.org/issues/show_bug.cgi?id=3763 (checkout > should re-use ra session for externals if possible) > > It's marked as "started", but I'm not sure if that's correct status at > this point. > > I'm cc'ing Ben Reser (who has the above issue assigned to him) and > Ivan Zhakov (who recenty wrote a patch for this). Perhaps they can > shed some light on the current state of "RA session caching". Thank you. > > As far as I remember from discussions during the Berlin Hackathon, > this was deemed too risky for 1.8, but perhaps within scope for 1.9. > > > > > All that said, is there a work around to convincing SVN to recreate the file > > without a full fledge update? > > No, I'm afraid I know of no workaround. > > Perhaps some further variation of what you're already trying (but I'm > sure you did already many trial and error attempts). I think a short term workaround would be to detect if an external file name path was changed and force a full update. For revision updates only, I believe I could still make this solution work (update local properties and iterate over externals that do not match, updating one by one). This will give us a huge performance boost for some of our cases. Thanks for the help! Dan
Re: How to change paths on an external file without a full update --depth infinity?
Ben Reser wrote on 08/11/2013 09:54:43 AM: > > On Sun, Aug 11, 2013 at 3:24 AM, Johan Corveleyn wrote: > > I'm cc'ing Ben Reser (who has the above issue assigned to him) and > > Ivan Zhakov (who recenty wrote a patch for this). Perhaps they can > > shed some light on the current state of "RA session caching". > > > > As far as I remember from discussions during the Berlin Hackathon, > > this was deemed too risky for 1.8, but perhaps within scope for 1.9. > > It's on my todo to review Ivan's patch. It seems like the best short > term solution and is basically what I had already intended to do. I'd be willing to provide some real world testing on this patch. I'm not sure I'd be able to get up to speed sufficiently on test models and the like, but would do what I can. This would be a huge boon for us as I know our place has two major toolsets in place that make extensive use of externals and the session caching (especially for satellite locations) performance issue is the number one frustration with the toolsets. Thanks, Dan
Re: Error processing externals E205011 and W155035 with simple update
Johan Corveleyn wrote on 08/11/2013 02:44:38 PM: > >> > Here's the quick recipe > >> > > >> > Create a file external in svn:externals and commit it and perform full > >> > update (foo.c rev 1). > >> > > Okay, with this information I can reproduce the issue (both with 1.7.9 > and with trunk@1512490): > > > I get it also when only using --ignore-externals (or --depth=empty), > i.e. it's not necessary to combine both options in the step before the > final update. > > Dan, can you please file an issue for this (referencing this thread)? > Thank you for reproducing this! I thought I was going crazy for a while... I've entered a new issue per your recommendations. http://subversion.tigris.org/issues/show_bug.cgi?id=4409 Thanks, Dan
Re: How to change paths on an external file without a full update --depth infinity?
Ben Reser wrote on 08/11/2013 09:54:43 AM: > From: Ben Reser > To: Johan Corveleyn > Cc: dlel...@rockwellcollins.com, "users@subversion.apache.org" > , Ivan Zhakov > Date: 08/11/2013 09:55 AM > Subject: Re: How to change paths on an external file without a full > update --depth infinity? > > On Sun, Aug 11, 2013 at 3:24 AM, Johan Corveleyn wrote: > > I'm cc'ing Ben Reser (who has the above issue assigned to him) and > > Ivan Zhakov (who recenty wrote a patch for this). Perhaps they can > > shed some light on the current state of "RA session caching". > > > > As far as I remember from discussions during the Berlin Hackathon, > > this was deemed too risky for 1.8, but perhaps within scope for 1.9. > > It's on my todo to review Ivan's patch. It seems like the best short > term solution and is basically what I had already intended to do. Hi, Will this look like a 1.8.x change in your mind? We are really sitting on pins and needles with excitement for this! I like Subversion - simple, powerful, and light weight. I've been a part of many trade studies over the years looking at how different groups choose the config mgmt tool of their choice (CC, CMVC, PVCS, Git, SVN, etc. have all been used). When it comes to product line strategy, the trade studies have typically given low marks to Subversion based on external performance (and certainly other open tools for other reasons too). Typically the results of these trade studies never get flowed back out as to why a tool wasn't chosen. I'm now seeing that this is unfortunate for open tools as they may not always see the desires of certain industries. In this case, as a new person to the SVN community, I admit I was surprised to see that the chatter within the SVN community didn't match the chatter I've been hearing for years among the various organizations within my company (we're fairly large ~20k ppl and of which the different organizations are largely independent and focused on vary different products yet we came to the same conclusions - externals are invaluable, yet painful). I believe that if we can improve external performance (speed and integration -- like handling externals when depth != infinity), not only would we help the current users of SVN that have come to accept this, but we would have a huge opportunity to get back on the radar of other users that have previously chosen other options. I completely appreciate the work you all do, SVN is a great tool. This is just some feedback from a user group that I'm concerned you may not have from properly. Going forward, I think a takeaway for me is that as industry uses open source tools and subsequently performs trade studies on those tools, its important to provide feedback to the open source community. Its not especially fair to use and not contribute in some form. I'm willing to help perform informal testing or in other tasks that could be useful. Thanks, Dan
Re: How to change paths on an external file without a full update --depth infinity?
Les Mikesell wrote on 08/14/2013 10:17:41 AM: > From: Les Mikesell > To: dlel...@rockwellcollins.com > Cc: Ben Reser , Ivan Zhakov , > Johan Corveleyn , "users@subversion.apache.org" > > Date: 08/14/2013 10:18 AM > Subject: Re: How to change paths on an external file without a full > update --depth infinity? > > On Wed, Aug 14, 2013 at 11:48 AM, wrote: > > > > I believe that if we can improve external performance (speed and integration > > -- like handling externals when depth != infinity), not only would we help > > the current users of SVN that have come to accept this, but we would have a > > huge opportunity to get back on the radar of other users that have > > previously chosen other options. > > I'm not sure that current SVN users accept problems with depth != > infinity as much as they arrange their layout so they don't have to do > that. What's a common use case for needing some disjoint arrangement > of components that you can't assemble with externals and normal > recursion? > This is a case of trying to improve performance on externals by only updating externals that have changed. Without connection caching, performing an external update over a WAN is a test of patience. For us, our repo is accessed over a WAN. Its not an issue for non-"file externals". For example, a WC of 1000 file externals will take over 15 minutes to update with zero changes but the same WC with no file externals (1000 normal files) takes 30 seconds tops. Keep in mind, no actual file revisions are being downloaded, just checked. We have been attempting to put in place a workaround to give us the benefit of file externals with a general algorithm of: update external table (svn ps svn:external -F tmpfile) commit external table (svn commit . --depth empty --ignore_externals) update the externals table for paranoia (svn update . --depth empty --ignore_externals) foreach external in svn:properties (compare svn:externals to svn info -v) if the local revision (svn info) is not the same as the external update the external (svn update foo.c) This has proven to be difficult to do since: a. a corruption issue was identified (bug 4409) b. we can't change a path on a file external (bug 4001) with out a full update (depth infinity) (old file gets deleted, new one doesn't get brought in) Either of these issues prevents doing a simple update of a file external or doing a current directory update (50 files is more bearable than a full 1000). That said, this attempted work around would not be needed if the connection caching can bring file external updates to something closer to the performance of regular files. Hope this helps, I'd be glad to explain in more detail how we implement this. I think we have a very successful strategy for our needs. Thanks Dan
Re: How to change paths on an external file without a full update --depth infinity?
Kevin M Radke/CedarRapids/RockwellCollins wrote on 08/14/2013 11:52:44 AM: > dlel...@rockwellcollins.com wrote on 08/14/2013 01:13:52 PM: > > > I'm not sure that current SVN users accept problems with depth != > > > infinity as much as they arrange their layout so they don't have to do > > > that. What's a common use case for needing some disjoint arrangement > > > of components that you can't assemble with externals and normal > > > recursion? > > > > > > > This is a case of trying to improve performance on externals by only > > updating externals that have changed. Without connection caching, > > performing an external update over a WAN is a test of patience. For > > us, our repo is accessed over a WAN. Its not an issue for non-"file > > externals". For example, a WC of 1000 file externals will take over > > 15 minutes to update with zero changes but the same WC with no file > > externals (1000 normal files) takes 30 seconds tops. Keep in mind, > > no actual file revisions are being downloaded, just checked. > > > > Even with connection caching, you will still be asking the > server a separate question for each external instead of one > question for the whole working copy. I think there will > always be a performance cost to using externals, and using > thousands is just asking for problems. We're ok with _some_ performance cost. Also, it might be valuable to the SVN community to understand other use cases when they make architectural decisions. I'm not sure its technically impossible to combine all the external requests by directory into a single transaction. Let's ask the community and developers what they think. > Designing a build > management system on top of subversion using only externals > is risky. I disagree, but we've had this conversation already. I'd be very welcome to try anything to help our performance.
Re: How to change paths on an external file without a full update --depth infinity?
Les Mikesell wrote on 08/14/2013 01:58:29 PM: > From: Les Mikesell > To: dlel...@rockwellcollins.com > Cc: kmra...@rockwellcollins.com, Ben Reser , Ivan > Zhakov , Johan Corveleyn , > "users@subversion.apache.org" > Date: 08/14/2013 01:58 PM > Subject: Re: How to change paths on an external file without a full > update --depth infinity? > > On Wed, Aug 14, 2013 at 2:04 PM, wrote: > > > > > >> Designing a build > >> management system on top of subversion using only externals > >> is risky. > > > > I disagree, but we've had this conversation already. I'd be very welcome to > > try anything to help our performance. > > Externals can be very useful to assemble mixed revisions of > separately-versioned components, but why do they have to be > file-level? Are there other tools involved in the process that do not > understand subdirectories or symlinks? Or just no natural groupings? > No natural groupings. In embedded, safety critical DO-178b products, we have very rigorous rules that essentially prevent one size fits all developments. Product line management on things like this become very hard and tend to create clone and owns of the code base as we move from one system to another. Requirements and verification (including structural coverage testing) is a beast. We did a study of similar projects and found that in unexpected amount of code is common that one wouldn't expect to be common. This ad hoc common code was larger than the amount we could actually purposely create as "designed for reuse" (libraries or directory level commonality). I'd imagine if anyone took a baseline and forked it, did a bunch of work and compared it back to the baseline, you'd be surprised what turns out as still being common (unless someone does a ton of style changes). Now try and imagine how to extract those random bits of code into groupings. It can be done, but in our world, you can't go off and willy nilly change and refactor code. Its nice to maintain a living pedigree with your baseline. Now, in our case, we do stuff for aircraft,... wouldn't it be nice to maintain living pedigrees with all similar models of aircraft? Fix an issue in one place and advertise it to all the others. File externals give you this. It fits very well into the embedded safety critical world in the "don't touch that code unless you have to" and "let's fix it once". Refactoring code in this world just can't happen as often as you'd like (its also a chance to reinject bugs). Hope this helps! Dan
Re: How to change paths on an external file without a full update --depth infinity?
Les Mikesell wrote on 08/14/2013 02:49:38 PM: > On Wed, Aug 14, 2013 at 4:23 PM, wrote: > > > > Now, in our case, we do stuff for aircraft,... wouldn't it be nice to > > maintain living pedigrees with all similar models of aircraft? Fix an issue > > in one place and advertise it to all the others. File externals give you > > this. It fits very well into the embedded safety critical world in the > > "don't touch that code unless you have to" and "let's fix it once". > > Refactoring code in this world just can't happen as often as you'dlike (its > > also a chance to reinject bugs). > > > > Hope this helps! > > So the point is to intentionally pull the HEAD revision of a whole > bunch of files together where each is located arbitrarily and can > change independently? I guess that's about the opposite of the way I > think of version control, so I can't suggest anything else. Are > there enough different 'top level' collections or a fast enough change > that you can't simply copy the right files into place instead of > having external references there? No HEAD revisions, all files are pegged. For example, if everyone is linked to a common file (foo.c, revision 20, project A "pedigree") and a bug is fixed, each project will see that a fix has been made. Each project has to make a decision: to remain on the current revision (no budget or schedule to update), update to the latest (if they have budget and schedule), or to "fork" (if they don't like the fix and have to implement a new feature). This can happen any time, not necessarily when the file is committed. Each new project inherits the externals from their baseline (they do not create new links or go out and search for reuse - that'd be a pain!) So until a project decides to "fork" a file, they see (but not automatically receive) all the changes made to that file. Nothing happens to your project without explicit action. Each file lives in a special area of the repository that identifies its pedigree. Code your project writes automatically becomes usable by any other project. This is the same approach Rational CMVC takes. Hope this helps, Dan
Re: How to change paths on an external file without a full update --depth infinity?
> > > > No HEAD revisions, all files are pegged. > > > > For example, if everyone is linked to a common file (foo.c, revision 20, > > project A "pedigree") and a bug is fixed, each project will see that a fix > > has been made. Each project has to make a decision: to remain on the > > current revision (no budget or schedule to update), update to the latest (if > > they have budget and schedule), or to "fork" (if they don't like the fix and > > have to implement a new feature). This can happen any time, not necessarily > > when the file is committed. > > Then what makes the collection of pegged externals easier to manage > than explicitly copying specific revisions into the top level where > the reference makes it appear and subsequently merging changes or > deleting files and copying in the updated version. Those operations > are relatively efficient with svn and copies carry along the revision > history. > > I don't follow how each project 'sees' that fixes have been made - you > shouldn't see that through a pegged external. We have a tool that mimics explorer. It queries the repo for the latest revision on each file external (yes a bit talky with the server). If they are not identical, it displays revision x of y. It also displays the pedigree (or "history"" in CMVC lingo). You can select each file, and graphically decide to "fork" to a different pedigree/history, or peg to a different revision. > > Each new project inherits the externals from their baseline (they do not > > create new links or go out and search for reuse - that'd be a pain!) So > > until a project decides to "fork" a file, they see (but not automatically > > receive) all the changes made to that file. Nothing happens to your project > > without explicit action. > > You'd get roughly the same effect with an 'svn cp' of a baseline - the > equivalent of a branch or tag even if you name it something else. I don't think this will do what we want. > > > Each file lives in a special area of the repository that identifies its > > pedigree. Code your project writes automatically becomes usable by any > > other project. > > None of that would need to change regardless of whether you copy a > revision into a different folder of reference it directly with an > external. As long as you aren't floating to the HEAD version you > have to do something to bump the revisions - why not just copy it in > the repo where remote revision checks will be fast? Once you copy, you break the link. If you were to make a change to the copy, no one else would then see it. You are correct, you could replicate all of this many different ways. When we did our study, externals (aside from the performance issue) was the best answer. I wish I could give a quick tour of the process/tool. It'd answer so many questions. Thanks, Dan
Re: How to change paths on an external file without a full update --depth infinity?
> > Once you copy, you break the link. If you were to make a change to the > > copy, no one else would then see it. > > No one else would see it with externals either, except that you > wrote a custom tool to analyze the externals, see if a newer > revision of the original exists, and show that to the user. If you > can do that with externals, you can do that with copies too. (Use > "svn log --stop-on-copy" to find out where the copy came from, then > see if there are newer revisions of that.) The challenge I then see on this is one of finding all instances of foo.c. If you have foo.c copied/forked fifty times to different projects, each of which has branched a couple of times, how do you programmatically find all different instances of foo.c (to let a developer choose which may be most appropriate?) If you have good ideas, I'm very open to listening. Also if you have to projects that both want foo.c and both have valid changes to make to the file, how does that get managed when they are copies? Its a trivial implementation when it is implemented as a file external. We also have instances where we purposely want multiple copies of the same exact file within the same project. We can effectively manage this through file externals to a structured "datastore" (AKA a set of folders within a repo). Regardless of where and how a team decides to structure their project, all files are neatly organized in this one section of the repo (that is considered taboo to directly interact with). The ability to have a specific file having many "copies" of itself and not care about its position within the repository is a powerful feature. I understand this may diverge a bit from SVN's core thoughts on CM, but if SVN can support odd variations to its use, it becomes an even more indispensable building block. Diversity in approaches is good. >From a feature perspective, externals are a very appropriate method to accomplish this (really a CM implementation of symlinks). If we're saying that externals from an implementation standpoint are not quite appropriate at this time, I get that argument. What is the general consensus as to where externals are on the roadmap? I may not convince the team that externals are really really useful (even if abused) in this application, but I'm hoping that the team does appreciate the general usefulness of externals and keeps maturing the feature. Thanks Dan
Re: How to change paths on an external file without a full update --depth infinity?
> > The challenge I then see on this is one of finding all instances of foo.c. > > If you have foo.c copied/forked fifty times to different projects, each of > > which has branched a couple of times, how do you programmatically find all > > different instances of foo.c (to let a developer choose which may be most > > appropriate?) If you have good ideas, I'm very open to listening. > > There is no difference in that question than finding where the > 'future' copies of a pegged external target went. You can only do > either if you have a convention for a canonical path. true (I believe). > > > Also if you have to projects that both want foo.c and both have valid > > changes to make to the file, how does that get managed when they are copies? > > Its a trivial implementation when it is implemented as a file external. > > How so? I assume you also have to handle cases either way: where both > projects want the same change and where both projects need different > changes - where typical svn users would have branches/tags to > distinguish them. But regardless of how you identify the target > file, there shouldn't be any effective difference between copying a > version into your directory or using a file external as long as you > don't modify it in place and commit it back - something your external > tool could track. We do want to modify in place. Copying back creates an additional step that is already managed quite well by SVN with externals. I don't want to duplicate something that already exists (I'll mess it up). There was some discussion on another thread about advancing a peg revision of an external when an external is committed. This would be a neat feature, though I completely understand why it would not be incorporated. We do this behind the scenes right now with very little work (post-commit script in TSVN gives us knowledge of what a user committed). > > > We also have instances where we purposely want multiple copies of the same > > exact file within the same project. We can effectively manage this through > > file externals to a structured "datastore" (AKA a set of folders within a > > repo). Regardless of where and how a team decides to structure their > > project, all files are neatly organized in this one section of the repo > > (that is considered taboo to directly interact with). The abilityto have a > > specific file having many "copies" of itself and not care about its position > > within the repository is a powerful feature. I understand this may diverge > > a bit from SVN's core thoughts on CM, but if SVN can support odd variations > > to its use, it becomes an even more indispensable building block. Diversity > > in approaches is good. > > Again, you get the history in a copy. You can tell if they are the > same. Or, on unix-like systems you can use symlinks to a canonical > copy within the project. We're not a unix-like system but that is what would work great (with the exception that you can't revision control symlinks, right?) > > > From a feature perspective, externals are a very appropriate method to > > accomplish this (really a CM implementation of symlinks). If we're saying > > that externals from an implementation standpoint are not quite appropriate > > at this time, I get that argument. What is the general consensus as to > > where externals are on the roadmap? > > I agree that externals are very useful, but most projects would use > them at subdirectory levels for component libraries where they work > nicely, not for thousands of individual file targets. Is there > really no natural grouping - perhaps even of sets of combinations that > have been tested together that you could usefully group in > release-tagged directories? The whole discovery we found is that most of our reuse occurred in unplanned ways (I'd imagine if you took two linux distros and compare files which changed and didn't change, it would be a huge collection of random files that aren't easily abstracted out. You might be able to do it once, but as each new distribution branches out, the commonality between each of them becomes impossible to form groupings on. > ...'just work' without special considerations. I'm not against better > performance, of course, but it makes sense to me to make pragmatic > design decisions for the same reasons you might avoid throwing > millions of files in one flat directory even in a non-versioned > scenario. Theoretically, you should be able to do that, but in > practice it isn't going to perform as well as something with better > structure. I'm not sure what a reasonable number of external files per folder is, but I'd think it'd be similar to a reasonable number of regular files would be. Two million is nuts, but 50 seems reasonable. The issue is that I'm currently forced to deal with not just the current directory, but the recursion on all nested directories (--depth infinity). If, as the subject of
Re: How to change paths on an external file without a full update --depth infinity?
> On Thu, Aug 15, 2013 at 12:39 PM, wrote: > > > >> But regardless of how you identify the target > >> file, there shouldn't be any effective difference between copying a > >> version into your directory or using a file external as long as you > >> don't modify it in place and commit it back - something your external > >> tool could track. > > > > We do want to modify in place. Copying back creates an additionalstep that > > is already managed quite well by SVN with externals. > > I've never done that with a file external - where does the commit go? It commits a new revision of what the file external pointed to - pretty handy. If you are pegged, it will not automagically update your pegged revision (as I'd expect), so unless you are on the HEAD or update your peg to what just committed, an update will revert your WC back to the pegged version. > > >> Again, you get the history in a copy. You can tell if they are the > >> same. Or, on unix-like systems you can use symlinks to a canonical > >> copy within the project. > > > > > > We're not a unix-like system but that is what would work great (with the > > exception that you can't revision control symlinks, right?) > > I think so, but the links and the target would be versioned > independently which might complicate your tracking. Yes, it would complicate things quite a bit and introduce areas for defects to get introduced. > > > > The whole discovery we found is that most of our reuse occurred inunplanned > > ways (I'd imagine if you took two linux distros and compare files which > > changed and didn't change, it would be a huge collection of random files > > that aren't easily abstracted out. You might be able to do it once, but as > > each new distribution branches out, the commonality between each of them > > becomes impossible to form groupings on. > > I was thinking of just adding an extra layer of grouping management > that would be versioned and able to be duplicated as much as > necessary. Suppose you made 10 directories and copied 100 files into > each with tagged versions of these directories for every combination > you need to access. Normally there would be natural groupings where > there is a common manager making decisions, etc., but for the moment > just consider it for performance. Within the repository, the copies > are cheap like symlinks - you could have a large number of > pre-arranged tagged choices. Then your top level project becomes 10 > directory-level externals instead of 1000 file externals. With more complexity comes more bugs and process missteps. We're really striving to keep things as simple as possible. We're fundamentally accepting of update times going from 2 seconds to 2 minutes. Its harder when 2 minutes becomes 20 minutes. > > > I'm not sure what a reasonable number of external files per folder is, but > > I'd think it'd be similar to a reasonable number of regular files would be. > > Two million is nuts, but 50 seems reasonable. > > Think of this in terms of client-server activity. With directory > level externals, the client can ask the server if anything under the > directory has newer revisions in one exchange and if it hasn't, you > are done. So what's reasonable is the amount of activity you want to > wait for. The whole discussion has centered on an attempted work around for the connection caching that doesn't currently occur for externals. If that can happen, I think we'd be very content. We're accepting of some performance issues. There was an XKCD a while ago that talked about how much time a task takes, how many times you do it and how much waste is created over a year. It was interesting (even if obvious if you thought about it). I think with connection caching we'd hit the sweet spot and working further would result in diminishing returns. This thread is an attempt a hopefully short-term work around this limitation. > Usually I'd consider the 'human' side of organization first, so if you > can come up with any groupings that could be done as copies into > tagged directories you might want to arrange them by the people/groups > who make the choices - and then the performance win would just be an > extra bonus. That's just it, we can't find (let alone maintain over time) any consistent groupings by function. Trying to create groupings other ways could confuse the developers, or if we try and hide the fact that we have an optimized backend of sorts, we then have to write more tool software (we don't like writing tools, we like writing embedded software). In the end, revision controlled symlinks are the best answer and file externals appear to be very close. And we're oh so close with file externals right now. Thanks Dan
Re: How to change paths on an external file without a full update --depth infinity?
> On Thu, Aug 15, 2013 at 2:03 PM, wrote: > > > >> With more complexity comes more bugs and process missteps. We're really > > striving to keep things as simple as possible. We're fundamentally > > accepting of update times going from 2 seconds to 2 minutes. Its harder > > when 2 minutes becomes 20 minutes. > > Are your build systems on the other side of the world from the > repository? The quick fix might be to reduce the latency one way or > another (move one of the pieces, use the wandisco distributed repo > approach, etc?) or automate with something like jenkins so you don't > have a person waiting. Yes, and that's a big contributor. Co-location helps significantly, but isn't an option in our case. I'll take a look at your suggestions. Thanks dan
Re: How to change paths on an external file without a full update --depth infinity?
> > > > It commits a new revision of what the file external pointed to - pretty > > handy. If you are pegged, it will not automagically update your pegged > > revision (as I'd expect), so unless you are on the HEAD or update your peg > > to what just committed, an update will revert your WC back to the pegged > > version. > > I'd actually expect it to be pretty confusing if you had multiple > people committing changes based from different back-rev pegged > references anywhere near the same time frame. Does your external > 'notify about new versions' tool help with that? Don't you get > conflicting changes that you'd want branches to help isolate? Yes, its obvious to users when they are not on the latest revision of the file, but they c/would still go through a merge process if need be. Its actually very straight-forward as we intentionally focused on targeting non-CM-guru folks.
Re: How to change paths on an external file without a full update --depth infinity?
Ben Reser wrote on 08/15/2013 02:57:23 PM: > On 8/15/13 1:05 PM, Les Mikesell wrote: > > I'd actually expect it to be pretty confusing if you had multiple > > people committing changes based from different back-rev pegged > > references anywhere near the same time frame. Does your external > > 'notify about new versions' tool help with that? Don't you get > > conflicting changes that you'd want branches to help isolate? > > The commit won't complete you'll get an out of date error. That's right, isn't it. It'd be no different than two folks trying to commit the same file around the same time, right (one would get an out of date error)? Thanks, Dan
Re: How to change paths on an external file without a full update --depth infinity?
> >> The commit won't complete you'll get an out of date error. > > > > That's right, isn't it. It'd be no different than two folks trying to > > commit the same file around the same time, right (one would get an out of > > date error)? > > Right, but when working on the trunk explicitly you'd expect to update > to accept others' changes often, or to branch to preserve and isolate > your differences. I don't see how either quite matches a model where > changes might be made based on multiple differing back-rev pinned > externals. What happens if two projects don't want to accept each > others' changes and need to commit their own? In a more typical > scenario they would be working on branch copies that could diverge, > but I think you lose that by forcing a canonical path for development > (as a tradeoff for knowing where the 'new' work is...). > If a project doesn't want to accept a change, they "fork" to a new "history". The tool does this with a svn cp and an update to the svn:externals property. They now lose sight of what the other project commits after that fork though. The backend of where the file is stored and how is masked by the tool. is essentially implemented as a folder. To the developer, they may know that their file is really a file external, but they don't treat it really any different from a normal file until it comes time to "fork". I try to differentiate "forking" as a pedigree/history from branching and the like. This system is essentially an implementation of Rational's CMVC history feature.
Re: How to change paths on an external file without a full update --depth infinity?
> > > > If a project doesn't want to accept a change, they "fork" to a new > > "history". The tool does this with a svn cp > > and an update to the svn:externals property. They now > > lose sight of what the other project commits after that fork though. The > > backend of where the file is stored and how is masked by the tool. > > is essentially implemented as a folder. To the developer, they > > may know that their file is really a file external, but they don't treat it > > really any different from a normal file until it comes time to "fork". I > > try to differentiate "forking" as a pedigree/history from branching and the > > like. > > > > This system is essentially an implementation of Rational's CMVC history > > feature. > > In subversion's view a copy is a branch so any distinction is strictly > your own convention. Likewise for tags, except that there is a > generally accepted convention of not committing changes after a tag > copy. Do you have additional conventions or tools to help users of > the pre-fork version know that it has branched?I don't think there > is a generic solution for that - subversion tracks copy-from history, > but not copy-to. No. There is no way to know who is using a fork you may have created or who has forked from someplace (short of scanning all projects of course). I'm not sure fork is the best name to give this concept, but we didn't want to choose branch or tag for obvious reasons
svn log for externals
Hello everyone, I thought I've seen discussion in the past on SVN behavior when doing an "svn log" on a WC with externals but my google skills seem weak. Currently svn log will display file changes and svn:externals property changes, but will not display changes made to the file pointed at by the external. For example, if a user commits a change to a file external, svn log will show that the properties reflects a new revision of the file is now being referenced, but will not see commit made to the file, even if it was done from the current WC. It would be a very useful feature to have svn log include the log of the file external(s) as if it were a formal vile within the WC (even if it required a switch --include-externals) to enable. Any thoughts on this behavior? Thanks, Dan
Programatically determine revision of commit
Hi, We're attempting to write some scripts to perform automatic commits for certain files under certain conditions. We wish to use the just committed revision number to perform some other logging operation. The --xml command line option on many of the other commands (info, status, etc.) are fantastic, but the commit command does not allow this. I'm currently attempting to parse the standard output of the commit message to grab it, but feel this is less than optima as it seems very error prone. Is there any consideration to add an --xml option to the commit command output or a better programmatic means to grab the commit revision? c:\Project_files\sandbox_v2>svn commit 1.txt -m "Checkin message". Sending1.txt Transmitting file data . Committed revision 3272. c:\Project_files\sandbox_v2> Thanks, Dan
Inconsistency with --xml for svn info and svn status ?
Hi, I think I found a subtle difference for outputting XML formatted data from svn info and status. I'm on Win7, SVN 1.8.5 (r1542147). svn status when generating tags does not seem to care about your current working directory (CWD) relative to the requested path. It always appears to return an absolute file path. svn info returns different path depending on if the WC path requested is within the EXACT spelling of the CWD. When your requested path is within the enclosing CWD, it will return relative paths and absolute otherwise. If the requested path spelling is equivalent (for Win systems any how), it will only return relative for exact spelling. Changing the case of a directory will change it to absolute. It appears there may be other ways to get relative vs absolute with info but I couldn't repeat reliably. It would be nice if SVN would always return an absolute path consistent with the status command. This would also solve the case where Win paths are case insensitive and equivalent paths return different results. It would also be good as generally when using --xml, you are using some automated tool or application and do not always know or control what the CWD is (especially for Win apps). Additionally, the path that the entry is relative to is not the WC but the CWD. This makes grabbing an XML file later and making sense of it impossible (since the CWD is not embedded within the XML file) -- see third example below. I'm not sure if this is considered operating as-design, and therefore an enhancement request, or if it is a bug. I appreciate your time! Thanks Dan Here's an example... This first call is using a case-different path from what the CWD is (note the project_files vs Project_files). It returned an absolute path. The second returned relative paths (exact spelling match). The third is showing the relative paths are not relative to a WC, but relative to the CWD. c:\Project_files\LocalSVN>svn info "c:\project_files\LocalSVN\trunk\info_test" --depth immediates --xml (Snipped) (Snipped) c:\Project_files\LocalSVN>svn info "c:\Project_files\LocalSVN\trunk\info_test" --depth immediates --xml (Snipped) (Snipped) c:\>svn info "c:\Project_files\LocalSVN\trunk\info_test" --depth immediates --xml (Snipped) (Snipped)