non-inheritable revisions in mergeinfo being later incorrectly marked as inheritable
Hello, This is my first time posting to this mailing list so apologies for any newbie mistakes. I couldn't find a bug report discussing this particular issue and while I'm fairly certain this isn't intended behavior, I wanted to post to users@ just in case there was some strange reasoning for why this is supposed to happen. I'll give the gist of what the issue is and then I'll post the sequence of commands I used to reproduce the entire issue, with some intermediate diffs to show whats going on. Essentially, I have a full checkout of trunk containing the following directory structure: file1.txt my-project/fileA.txt not-my-project/fileA.txt Then we have a copy of trunk, called "my-branch" which has everything checked out except for not-my-project. We make a sequence of commits in trunks, each of which is followed by a sync merge from trunk to my-branch. The first commit changes "file1.txt" and the mergeinfo reflects a full merge (good). The next two commits involve changes to not-my-project, and the mergeinfo of my-branch reflects these revisions as non-inheritable (so, partially merged. also good). Finally, we make a commit to "file1.txt" again, but when merging that over, the mergeinfo in my-branch suddenly reverse-merges the non-inheritable ranges and merges them as inheritable, despite the fact that these revisions have not had their changes brought in (we still don't have not-my-project checked out). The consequence is that SVN now thinks that my-branch has had not-my-project updated and a subsequent reintegrate merge from my-branch to trunk reverses all of the changes made in trunk to match what it looked like on the creation of my-branch. It does this with no complaints. Here is all of the steps I did to recreate that: Steps for creating trunk and my-branch: BQ_BEGIN awilton@metompkin:~/svnplayground/bug_repo_wc$ svn co file:///home/awilton/svnplayground/repo/bug_repo . Checked out revision 0. awilton@metompkin:~/svnplayground/bug_repo_wc$ svn mkdir trunk A trunk awilton@metompkin:~/svnplayground/bug_repo_wc$ svn mkdir branches A branches awilton@metompkin:~/svnplayground/bug_repo_wc$ svn commit -m "Created trunk and branches." Adding branches Adding trunk Committing transaction... Committed revision 1. awilton@metompkin:~/svnplayground/bug_repo_wc$ svn up Updating '.': At revision 1. awilton@metompkin:~/svnplayground/bug_repo_wc$ cd trunk awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ echo file1 > file1.txt awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ svn add file1.txt A file1.txt awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ svn mkdir my-project A my-project awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ svn mkdir not-my-project A not-my-project awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ echo my file > my-project/fileA.txt awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ echo not my file > not-my-project/fileA.txt awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ svn add my-project/fileA.txt A my-project/fileA.txt awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ svn add not-my-project/fileA.txt A not-my-project/fileA.txt awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ svn stat A file1.txt A my-project A my-project/fileA.txt A not-my-project A not-my-project/fileA.txt awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ svn commit -m "Set up trunk." Adding file1.txt Adding my-project Adding my-project/fileA.txt Adding not-my-project Adding not-my-project/fileA.txt Transmitting file data ...done Committing transaction... Committed revision 2. awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ cd .. awilton@metompkin:~/svnplayground/bug_repo_wc$ svn up Updating '.': At revision 2. awilton@metompkin:~/svnplayground/bug_repo_wc$ svn cp ^/trunk ^/branches/my-branch -m "Created my-branch from trunk" Committing transaction... Committed revision 3. awilton@metompkin:~/svnplayground/bug_repo_wc$ cd branches/ awilton@metompkin:~/svnplayground/bug_repo_wc/branches$ svn up my-branch --set-depth=files Updating 'my-branch': A my-branch A my-branch/file1.txt Updated to revision 3. awilton@metompkin:~/svnplayground/bug_repo_wc/branches$ cd my-branch/ awilton@metompkin:~/svnplayground/bug_repo_wc/branches/my-branch$ svn up my-project --set-depth=infinity Updating 'my-project': A my-project A my-project/fileA.txt Updated to revision 3. awilton@metompkin:~/svnplayground/bug_repo_wc/branches/my-branch$ ls file1.txt my-project awilton@metompkin:~/svnplayground/bug_repo_wc/branches/my-branch$ ls my-project/ fileA.txt BQ_END Change 1 to trunk, changing a shared file, "file1.txt": BQ_BEGIN awilton@metompkin:~/svnplayground/bug_repo_wc/branches/my-branch$ cd ../../trunk/ awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ echo change 1 to file1 > file1.txt awilton@metompkin:~/svnplayground/bug_repo_wc/trunk$ svn commit -m "Changed file1 in t
Re: non-inheritable revisions in mergeinfo being later incorrectly marked as inheritable
On Fri, Apr 24, 2020 at 10:36 AM Alex Wilton wrote: > > Hello, > > This is my first time posting to this mailing list so apologies for any > newbie mistakes. I couldn't find a bug report discussing this particular > issue and while I'm fairly certain this isn't intended behavior, I wanted to > post to users@ just in case there was some strange reasoning for why this is > supposed to happen. > > I'll give the gist of what the issue is and then I'll post the sequence of > commands I used to reproduce the entire issue, with some intermediate diffs > to show whats going on. > > Essentially, I have a full checkout of trunk containing the following > directory structure: > > file1.txt > my-project/fileA.txt > not-my-project/fileA.txt > > Then we have a copy of trunk, called "my-branch" which has everything checked > out except for not-my-project. > We make a sequence of commits in trunks, each of which is followed by a sync > merge from trunk to my-branch. The first commit changes "file1.txt" and the > mergeinfo reflects a full merge (good). The next two commits involve changes > to not-my-project, and the mergeinfo of my-branch reflects these revisions as > non-inheritable (so, partially merged. also good). Finally, we make a commit > to "file1.txt" again, but when merging that over, the mergeinfo in my-branch > suddenly reverse-merges the non-inheritable ranges and merges them as > inheritable, despite the fact that these revisions have not had their changes > brought in (we still don't have not-my-project checked out). The consequence > is that SVN now thinks that my-branch has had not-my-project updated and a > subsequent reintegrate merge from my-branch to trunk reverses all of the > changes made in trunk to match what it looked like on the creation of > my-branch. It does this with no complaints. Here is all of the steps I did to > recreate that: > Thank you for your analysis and detailed reproduction steps. Hopefully someone with more knowledge than me about the inner workings of mergeinfo will be able to chime in here, but my immediate thoughts in the meantime... I wonder if the problem still manifests if, in your first steps for setting up trunk and branch, you were to change this: $ cd branches/ $ svn up my-branch --set-depth=files to this: $ cd branches/ $ svn up my-branch --set-depth=immediates My thinking is that because the 'not-my-project' directory is not present at all in the branch working copy, perhaps svn isn't updating mergeinfo for it when it should, in one of the steps along the way, eventually culminating in the reversal of 'not-my-project' files on trunk. At the moment I can't experiment with this further, but I'll try to do some testing and digging through the mailing list archives over the weekend. Thanks, Nathan
Who else is using SVN for large-binary-asset storage?
Are there other companies out there using SVN for large-binary-blob storage? I'm wondering if it might be possible to put together a mini-consortium of companies to fund the completion of Issue #525: https://issues.apache.org/jira/browse/SVN-525 "allow working copies without text-base/" Our company keeps medium-large (say, ~5GB) binary blobs under version control in a dedicated Subversion repository, and it works quite well. Subversion can handle files of that size just fine, and it enables us to do path-based authorization (yay) and partial-tree checkouts. [1] But the presence of text-base files in .svn/pristine/ is a real downer :-). The text-base files are mostly pointless in this case, and they double the client-side disk space usage. There is no useful diff between two revisions of these binary blobs: there's no human-readable diff *and* there's rarely any machine-useable diff either (e.g., for reducing network time when receiving an update or committing a new revision). So the only benefit from the text-base files is to make 'svn revert' faster. We'd be happy to have 'svn revert' re-fetch the file from the repository if it meant we could reduce our storage cost on the client side by half. (Plus you'd only lose local-revert capability on files where you know you don't need it, since presumably the no-text-base behavior would be optional per file and controlled via an 'svn:no-pristine' property or something like that.) Is anyone else in a similar situation? If we join forces, we could probably fund one or more Subversion developers to finally get Issue #525 done. I'd be happy to do the organizing (I'm still reasonably familiar with Subversion development and who does what, though I haven't been an active developer in the project in many years). Please CC me on any replies, as I'm not subscribed to users@. Best regards, -Karl [1] We investigated using Git too, but, though Git good for many things, it is not well-suited for this particular job. The Git Large-File Storage extension (https://git-lfs.github.com/) doesn't address most of our needs either; it's solving a different problem, I guess.
RE: EXTERNAL: Who else is using SVN for large-binary-asset storage?
Hello everyone, The answer is "yes". I have come across investment banks that store boost releases in subversion. Sometimes it's a full boost release and that can be quite large. They take a long time to checkout but at the time it seemed better than the alternatives available at the time, e.g. placing the artifacts on an SFTP site or Windows share (ugh). -Original Message- From: Karl Fogel Sent: 24 April 2020 17:40 To: Subversion Users Subject: EXTERNAL: Who else is using SVN for large-binary-asset storage? Are there other companies out there using SVN for large-binary-blob storage? I'm wondering if it might be possible to put together a mini-consortium of companies to fund the completion of Issue #525: https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSVN-525&data=02%7C01%7CAndrew.Marlow%40fisglobal.com%7C6729a1c6e8434e58ae4b08d7e86e7f0d%7Ce3ff91d834c84b15a0b418910a6ac575%7C0%7C0%7C637233433584704974&sdata=WDAoL3RF5rOb0nx7Ig1wFIZkrisKSw%2BgKC1nGZ2GaPM%3D&reserved=0 "allow working copies without text-base/" Our company keeps medium-large (say, ~5GB) binary blobs under version control in a dedicated Subversion repository, and it works quite well. Subversion can handle files of that size just fine, and it enables us to do path-based authorization (yay) and partial-tree checkouts. [1] But the presence of text-base files in .svn/pristine/ is a real downer :-). The text-base files are mostly pointless in this case, and they double the client-side disk space usage. There is no useful diff between two revisions of these binary blobs: there's no human-readable diff *and* there's rarely any machine-useable diff either (e.g., for reducing network time when receiving an update or committing a new revision). So the only benefit from the text-base files is to make 'svn revert' faster. We'd be happy to have 'svn revert' re-fetch the file from the repository if it meant we could reduce our storage cost on the client side by half. (Plus you'd only lose local-revert capability on files where you know you don't need it, since presumably the no-text-base behavior would be optional per file and controlled via an 'svn:no-pristine' property or something like that.) Is anyone else in a similar situation? If we join forces, we could probably fund one or more Subversion developers to finally get Issue #525 done. I'd be happy to do the organizing (I'm still reasonably familiar with Subversion development and who does what, though I haven't been an active developer in the project in many years). Please CC me on any replies, as I'm not subscribed to users@. Best regards, -Karl [1] We investigated using Git too, but, though Git good for many things, it is not well-suited for this particular job. The Git Large-File Storage extension (https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit-lfs.github.com%2F&data=02%7C01%7CAndrew.Marlow%40fisglobal.com%7C6729a1c6e8434e58ae4b08d7e86e7f0d%7Ce3ff91d834c84b15a0b418910a6ac575%7C0%7C0%7C637233433584714968&sdata=XLfHamWk4don%2F1w77lC8zcC85ORUS%2FCnkuB8ft53XGs%3D&reserved=0) doesn't address most of our needs either; it's solving a different problem, I guess. The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. FIS is a trading name of the following companies: Advanced Portfolio Technologies Ltd (No: 6312142) | Clear2Pay Limited (No: 5792457) | Decalog (UK) Limited (No: 2567370) | FIS Apex (International) Limited (No: 260) | FIS Apex (UK) Limited (No. 3573008) | FIS Consulting Services (UK) Limited (No: 2486794) | FIS Derivatives Utility Services (UK) Limited (No: 9398140) | FIS Energy Solutions Limited (No: 1889028) | FIS Global Execution Services Limited (No. 3127109) | FIS Global Trading (UK) Limited (No: 2523114) | FIS Investment Systems (UK) Limited (No: 1366010) | FIS Sherwood Systems Group Limited (No: 982833) | FIS Systems Limited (No: 1937159) | FIS Treasury Systems (Europe) Limited (No: 2624209) | FIS Treasury Systems (UK) Limited (No: 2893376) | GL Settle Limited (No: 2396127) | Integrity Treasury Solutions Europe Limited (No: 3289271) | Monis Software Limited (No: 2333925) | Reech Capital Limited (No: 3649490) | Solutions Plus Consulting Services Limited (No: 3839487) | Valuelink Information Services Limited (No: 3827424) all registered in England & Wales with their registered office at 25 Canada Square, London E14 5LQ | FIS Global Execution Services Limited is authorised and regulated by the Financial Conduct Authority | Certegy Card Services Limited (No: 3517639) | Cer
Re: Who else is using SVN for large-binary-asset storage?
On Fri, Apr 24, 2020 at 12:42 PM Karl Fogel wrote: > Are there other companies out there using SVN for large-binary-blob > storage? > > I'm wondering if it might be possible to put together a mini-consortium of > companies to fund the completion of Issue #525: > > https://issues.apache.org/jira/browse/SVN-525 > "allow working copies without text-base/" > > Our company keeps medium-large (say, ~5GB) binary blobs under version > control in a dedicated Subversion repository, and it works quite well. > Subversion can handle files of that size just fine, and it enables us to do > path-based authorization (yay) and partial-tree checkouts. [1] > > But the presence of text-base files in .svn/pristine/ is a real downer > :-). The text-base files are mostly pointless in this case, and they > double the client-side disk space usage. > > There is no useful diff between two revisions of these binary blobs: > there's no human-readable diff *and* there's rarely any machine-useable > diff either (e.g., for reducing network time when receiving an update or > committing a new revision). So the only benefit from the text-base files > is to make 'svn revert' faster. We'd be happy to have 'svn revert' > re-fetch the file from the repository if it meant we could reduce our > storage cost on the client side by half. > > (Plus you'd only lose local-revert capability on files where you know you > don't need it, since presumably the no-text-base behavior would be optional > per file and controlled via an 'svn:no-pristine' property or something like > that.) > > Is anyone else in a similar situation? If we join forces, we could > probably fund one or more Subversion developers to finally get Issue #525 > done. I'd be happy to do the organizing (I'm still reasonably familiar > with Subversion development and who does what, though I haven't been an > active developer in the project in many years). > > Please CC me on any replies, as I'm not subscribed to users@. > > Best regards, > -Karl > > [1] We investigated using Git too, but, though Git good for many things, > it is not well-suited for this particular job. The Git Large-File Storage > extension (https://git-lfs.github.com/) doesn't address most of our needs > either; it's solving a different problem, I guess. > I think this would be a good idea in that it might be one of the last remaining niches where SVN is a better tool for the job than a DVCS. I do not think I could contribute though. I just wanted to throw another item on the pile. I recall an old thread (have not been able to find it) where it was shown that a massive performance win on large binary blobs would be if we could skip all of the xdelta stuff and just stream the binary. If I recall correctly, you can even see and demo this today using WebDAV and just doing a PUT or whatever is right request with the entire file. The server already knows how to handle it and store the file the same as it would if it had come via a SVN client. I think there were some complications with how svndiff0/svndiff1 etc are expected by a client, but if there were some way to have a property on a file that caused us to skip all of this, including storing the extra pristine copy, it could be a big win for managing large binaries with SVN. It seems like we could make revert fetch the file from the server again to restore a binary. If I can find any of those old threads I will share them. So far the only one I found was about how using a larger xdelta window size could give better compression, but the thread I recall was about not doing it at all. It also assume that the xdelta is of no real value because it does not shrink the amount of bytes that have to be transferred. -- Thanks Mark Phippard
Re: EXTERNAL: Who else is using SVN for large-binary-asset storage?
On 24 Apr 2020, Marlow, Andrew wrote:
>Hello everyone,
>
>The answer is "yes". I have come across investment banks that store
>boost releases in subversion. Sometimes it's a full boost release and
>that can be quite large. They take a long time to checkout but at the
>time it seemed better than the alternatives available at the time,
>e.g. placing the artifacts on an SFTP site or Windows share (ugh).
Ah -- very interesting, and not at all surprising! We've also enountered that
kind of usage at similar institutions, from talking to our clients.
If you're comfortable doing so, please feel free to introduce me privately to
whoever you think would be the right people at those investment banks. I'll
try to gather interest as efficiently as possible, and if we have enough
interest we can make something happen. I think those conversations have to at
least start in private, since many of those institutions are leery about
sharing *any* operational details, so I'd start from an "assume confidentiality
until explicitly told otherwise" basis.
In case users@ masks my email address: it's kfogel {_AT_}
opentechstrategies.com.
Best regards,
-Karl
>-Original Message-
>From: Karl Fogel
>Sent: 24 April 2020 17:40
>To: Subversion Users
>Subject: EXTERNAL: Who else is using SVN for large-binary-asset storage?
>
>Are there other companies out there using SVN for large-binary-blob storage?
>
>I'm wondering if it might be possible to put together a mini-consortium of
>companies to fund the completion of Issue #525:
>
>
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FSVN-525&data=02%7C01%7CAndrew.Marlow%40fisglobal.com%7C6729a1c6e8434e58ae4b08d7e86e7f0d%7Ce3ff91d834c84b15a0b418910a6ac575%7C0%7C0%7C637233433584704974&sdata=WDAoL3RF5rOb0nx7Ig1wFIZkrisKSw%2BgKC1nGZ2GaPM%3D&reserved=0
> "allow working copies without text-base/"
>
>Our company keeps medium-large (say, ~5GB) binary blobs under version
>control in a dedicated Subversion repository, and it works quite well.
>Subversion can handle files of that size just fine, and it enables us
>to do path-based authorization (yay) and partial-tree checkouts. [1]
>
>But the presence of text-base files in .svn/pristine/ is a real downer :-).
>The text-base files are mostly pointless in this case, and they double the
>client-side disk space usage.
>
>There is no useful diff between two revisions of these binary blobs:
>there's no human-readable diff *and* there's rarely any
>machine-useable diff either (e.g., for reducing network time when
>receiving an update or committing a new revision). So the only
>benefit from the text-base files is to make 'svn revert' faster. We'd
>be happy to have 'svn revert' re-fetch the file from the repository if
>it meant we could reduce our storage cost on the client side by half.
>
>(Plus you'd only lose local-revert capability on files where you know
>you don't need it, since presumably the no-text-base behavior would be
>optional per file and controlled via an 'svn:no-pristine' property or
>something like that.)
>
>Is anyone else in a similar situation? If we join forces, we could
>probably fund one or more Subversion developers to finally get Issue
>#525 done. I'd be happy to do the organizing (I'm still reasonably
>familiar with Subversion development and who does what, though I
>haven't been an active developer in the project in many years).
>
>Please CC me on any replies, as I'm not subscribed to users@.
>
>Best regards,
>-Karl
>
>[1] We investigated using Git too, but, though Git good for many
>things, it is not well-suited for this particular job. The Git
>Large-File Storage extension
>(https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit-lfs.github.com%2F&data=02%7C01%7CAndrew.Marlow%40fisglobal.com%7C6729a1c6e8434e58ae4b08d7e86e7f0d%7Ce3ff91d834c84b15a0b418910a6ac575%7C0%7C0%7C637233433584714968&sdata=XLfHamWk4don%2F1w77lC8zcC85ORUS%2FCnkuB8ft53XGs%3D&reserved=0)
>doesn't address most of our needs either; it's solving a different
>problem, I guess.
>The information contained in this message is proprietary and/or
>confidential. If you are not the intended recipient, please: (i)
>delete the message and all copies; (ii) do not disclose, distribute or
>use the message in any manner; and (iii) notify the sender
>immediately. In addition, please be aware that any message addressed
>to our domain is subject to archiving and review by persons other than
>the intended recipient. FIS is a trading name of the following
>companies: Advanced Portfolio Technologies Ltd (No: 6312142) |
>Clear2Pay Limited (No: 5792457) | Decalog (UK) Limited (No: 2567370) |
>FIS Apex (International) Limited (No: 260) | FIS Apex (UK) Limited
>(No. 3573008) | FIS Consulting Services (UK) Limited (No: 2486794) |
>FIS Derivatives Utility Services (UK) Limited (No: 9398140) | FIS
>Energy Solutions Limited (No: 1889028) | FIS Global Execution Services
>Limited (No. 3127109)
Re: Who else is using SVN for large-binary-asset storage?
On 24 Apr 2020, Mark Phippard wrote: >I think this would be a good idea in that it might be one of the last >remaining niches where SVN is a better tool for the job than a DVCS. >I do not think I could contribute though. > >I just wanted to throw another item on the pile. I recall an old >thread (have not been able to find it) where it was shown that a >massive performance win on large binary blobs would be if we could >skip all of the xdelta stuff and just stream the binary. If I recall >correctly, you can even see and demo this today using WebDAV and just >doing a PUT or whatever is right request with the entire file. The >server already knows how to handle it and store the file the same as >it would if it had come via a SVN client. I think there were some >complications with how svndiff0/svndiff1 etc are expected by a >client, but if there were some way to have a property on a file that >caused us to skip all of this, including storing the extra pristine >copy, it could be a big win for managing large binaries with SVN. > >It seems like we could make revert fetch the file from the server >again to restore a binary. > >If I can find any of those old threads I will share them. So far the >only one I found was about how using a larger xdelta window size >could give better compression, but the thread I recall was about not >doing it at all. It also assume that the xdelta is of no real value >because it does not shrink the amount of bytes that have to be >transferred. Ah, thanks for this reminder! I also recall those results (and I guess they're not surprising). I'll make sure we keep it in mind if this project happens. If you happen to dig up any of the old threads, that'd be great, but even if you don't, the above information is enough for a developer to know the possibility exists. Best regards, -Karl
